readium-shared/org.readium.r2.shared.util.tokenizer/NaiveTextTokenizer

NaiveTextTokenizer

class NaiveTextTokenizer(unit: TextUnit) : Tokenizer<String, IntRange>

A naive Tokenizer relying on java.text.BreakIterator to split the content.

Use IcuTextTokenizer for better results.

Constructors

NaiveTextTokenizer

constructor(unit: TextUnit)

Functions

open override fun tokenize(data: String): List<IntRange>