Cheshire3 Object Model - TokenMerger¶
API¶
- class cheshire3.baseObjects.TokenMerger(session, config, parent=None)[source]¶
A TokenMerger merges identical tokens and returns a hash.
A TokenMerger takes an ordered list of tokens (i.e. as produced by a Tokenizer) and merges them into a hash. This might involve merging multiple tokens per key, while maintaining frequency, proximity information etc.
One or more Normalizers may occur in the processing chain between a Tokenizer and TokenMerger in order to reduce dimensionality of terms.
Implementations¶
The following implementations are included in the distribution by default:
- class cheshire3.tokenMerger.SequenceRangeTokenMerger(session, config, parent)[source]¶
Merges tokens into a range for use in RangeIndexes.
Assumes that we’ve tokenized a single value into pairs, which need to be concatenated into ranges.