Package-level declarations

A default cluster TextTokenizer taking advantage of the best capabilities of each Android version.

Implementation of a TextTokenizer using ICU components to perform the actual tokenization while taking into account languages specificities.

A naive Tokenizer relying on java.text.BreakIterator to split the content.

A tokenizer splitting a String into range tokens (e.g. words, sentences, etc.).

A text token unit which can be used with a TextTokenizer.

A tokenizer splits a piece of data D into a list of T tokens.