readium-shared/org.readium.r2.shared.util.tokenizer/IcuTextTokenizer

IcuTextTokenizer

@RequiresApi(value = 24)
class IcuTextTokenizer(language: Language?, unit: TextUnit) : Tokenizer<String, IntRange>

Implementation of a TextTokenizer using ICU components to perform the actual tokenization while taking into account languages specificities.

Constructors

Link copied to clipboard
constructor(language: Language?, unit: TextUnit)

Functions

Link copied to clipboard
open override fun tokenize(data: String): List<IntRange>
Generated by Dokka
© 2026 Copyright