speech

Utterance Extraction

extractUtterances walks a Guided Navigation tree into a flat, ordered list of ReadiumSpeechUtterances ready for playback.

import { parseMarkup } from "@readium/guided-navigation";
import { extractUtterances } from "@readium/speech";

const gnd = parseMarkup(`<p lang="en">It was a dark and stormy night.</p>`);

await extractUtterances(gnd, { format: "plain" });
// [{ language: "en", plain: "It was a dark and stormy night." }]

Takes GuidedNavigationObject[] (parseMarkup()’s return / GuidedNavigationDocument.guided), not a wrapped GuidedNavigationDocument.

interface ReadiumSpeechUtterance {
  id?: string;
  plain?: string;
  ssml?: string;
  language?: string; // BCP 47
  voice?: string; // never set by extraction, see Playback.md
  locate?: LocatorOptions; // Decoded from the source node's textref — spread into createLocator()/decorate(), see GuidedNavigation.md
  offsets?: UtteranceOffset[]; // Ranges of plain/ssml backed by real source text, each with its own locate
}

interface UtteranceOffset {
  start: number;
  end: number;
  locate: LocatorOptions;
}

Some roles get a synthesized navigational contextualization spoken around their content (entering/leaving a table, a pagebreak label…) — see defaultContextualizations, sourced from locales/en.json. A synthesized label/announcement carries no offsets at all — locate is still safe for element-scoped highlighting, but there’s no real source text to search for.

Options

interface ExtractUtterancesOptions {
  format: "plain" | "ssml";
  skip?: GndRole[];
  contextualize?: GndRole[];
  contextualizationLocale?: string;
  contextualization?: ContextualizationOptions;
  language?: "none" | "block-level" | "always";
  contentLanguage?: string;
  languageOverrides?: Record<string, string>;
  interruptions?: Partial<Record<GndRole, "never" | "inline" | "after">>;
  segmentation?: SegmentationOptions;
  substitutions?: SubstitutionTable;
}

interface ContextualizationOptions {
  contextualizations?: Contextualizations;
  shapes?: Partial<Record<GndRole, "inline" | "block">>;
  params?: (role: GndRole, node: GuidedNavigationObject) => Record<string, string> | undefined;
}

interface SegmentationOptions {
  mode?: "structure" | "sentence"; // default "structure"
  suppressions?: Record<string, string[]>; // per-language sentence-ending exceptions, keyed like `language`
  segmenter?: SentenceSegmenter; // replaces the built-in Intl.Segmenter-based sentence splitter
}

type SentenceSegmenter = (
  language: string,
  text: string,
  customSuppressions?: string[]
) => Promise<SentenceBoundary[]>;

interface SentenceBoundary {
  text: string;
  start: number;
  end: number; // reaches through trailing whitespace to the next sentence
  contentEnd: number; // start + text.length, no trailing separator — offsets use this, not `end`
}

type SubstitutionRule = string | { pattern: RegExp; replace: string | ((...match: string[]) => string) };
type SubstitutionTable = Record<string, SubstitutionRule>;

Quick reference:

Option	Default	Purpose
`format`	`"plain"`	which field (`plain`/`ssml`) every utterance carries
`skip`	none skipped	drop a role and its subtree entirely
`contextualize`	none contextualized	which roles get a spoken announcement
`contextualizationLocale`	`"en"`	which shipped wording/plural-rules catalog to start from
`contextualization.contextualizations`	shipped catalog	reword or add announcement text, per role, on top of that catalog
`contextualization.shapes`	per the catalog	which of a role’s shapes (single line vs. bounded pair) is spoken — the piece verbosity levels drive to make a role’s announcement vary by level
`contextualization.params`	none	supply a placeholder value the extractor has no built-in source for
`language`	`"block-level"`	how a node’s own inline-language spans render
`contentLanguage`	none	the language of content without `lang`/`xml:lang`
`languageOverrides`	none	the language content declared in a given language is spoken in
`interruptions`	each `"after"`	whether a note, page break, abbreviation expansion, definition or bibliography entry is read never, inline or after
`segmentation.mode`	`"structure"`	one utterance per structural unit, or split at real sentence boundaries
`segmentation.reconstruct`	`false`	in sentence mode, join a sentence split across blocks (fixed layout)
`segmentation.suppressions`	none	per-language abbreviations (e.g. `"d."`) that sentence mode won’t treat as endings
`segmentation.segmenter`	built-in `Intl.Segmenter`-based splitter	swap in a different sentence segmenter
`substitutions`	`builtInSubstitutions`	ASCII imitations of Unicode symbols (`"1/2"`, `"(c)"`, `"100deg"`) to rewrite before speaking

`format`

Picks the one field every utterance in the result carries, so a consumer never has to check per-utterance which of plain/ssml is populated. Whichever a GuidedNavigationObject doesn’t natively have is synthesized (plain → escaped ssml; ssml → tags stripped to plain).

`skip`

Drops a role and its whole subtree — content and announcement both — from the output. skippableRoles export is the roles.md skippable set, minus the roles interruptions controls, which skip ignores.

A role a more specific one on the same node drops isn’t skipped either: skipping aside keeps <aside epub:type="footnote">, which skip: ["footnote"] drops instead. A pullquote or a cover drops its aside/blockquote or image this way only when contextualized.

`contextualize`

Which roles get an announcement. The wording comes from the shipped catalog by default — no need to supply your own contextualizations entry unless you want to change it (see below); a role with no entry at all, shipped or custom, just stays silent. Unlike skip, the underlying content still plays either way — only the extra announcement is gated by this.

// <p>...in the middle <span epub:type="pagebreak" title="5"/> of a sentence.</p>
await extractUtterances(gnd, { format: "plain" });
// [{ plain: "Page 4" }, { language: "en", plain: "...in the middle of a sentence." }, { plain: "Page 5" }]

await extractUtterances(gnd, { format: "plain", contextualize: ["pagebreak"] });
// [{ plain: "Pagebreak. Page 4." }, { language: "en", plain: "...in the middle of a sentence." }, { plain: "Pagebreak. Page 5." }]

await extractUtterances(gnd, { format: "plain", skip: ["pagebreak"] });
// [{ language: "en", plain: "...in the middle of a sentence." }]

Contextualization

contextualizationLocale and contextualization are unrelated to each other:

contextualizationLocale just retrieves a default shipped JSON catalog — it’s a data-source switch, nothing more. Most consumers only need this one. Every announcement carries that catalog’s locale as its language (English when the requested locale isn’t shipped), unless language is "none", which gives it contentLanguage; languageOverrides doesn’t apply to the catalog’s. Merged with content in the same primary language, it takes the content’s; in another language, it’s wrapped in a <lang> tag in SSML, and stays its own utterance in plain text.
contextualization (contextualizations/shapes/params) overrides whatever catalog ends up loaded — these three depend on each other, not on which locale you picked, which is why they’re grouped into one object instead of three flat top-level options:
- contextualizations rewords or extends the loaded catalog, for any subset of roles.
- shapes picks, for a role whose entry (after that reword) has both forms, which one actually gets spoken.
- params supplies data for a placeholder that wording references but the extractor has no built-in source for.

`contextualizationLocale`

Which shipped catalog to use (wording + plural rules). Falls back to "en" if the given locale isn’t shipped. Distinct from language further below, which governs the content’s own inline spans, not the catalog’s.

`contextualization.contextualizations`

Override or extend the shipped wording catalog for any subset of roles, at any depth (e.g. just table.block.start) — merged over the locale’s default. See Contextualization catalog below for its shape.

// Reword one existing role...
await extractUtterances(gnd, {
  format: "plain",
  contextualize: ["figure"],
  contextualization: { contextualizations: { figure: { inline: "Illustration: {{ description }}" } } },
});

// ...or add wording for a role the shipped catalog has no entry for at all,
// e.g. a custom GND extension role not in the spec.
await extractUtterances(gnd, {
  format: "plain",
  contextualize: ["sidebar"],
  contextualization: { contextualizations: { sidebar: { block: { start: "Sidebar.", end: "End of sidebar." } } } },
});

`contextualization.shapes`

Exists so a role’s announcement can be a single upfront line ("inline") or a bounded pair ("block", start before the content/end after) — for a role whose catalog entry defines both forms. A role with just one form always uses that one; this field has nothing to switch for it.

At the raw extractUtterances level this is just the resolved choice for one call — a flat { role: "inline" | "block" } map, not something that varies on its own. A caller that wants a role’s shape to vary by context (e.g. by verbosity level) re-resolves this map and re-calls extractUtterances each time that context changes — which is exactly what ReadiumSpeechNavigator does internally, see Playback.

Define both forms for a role of your own via contextualization.contextualizations, then pick which one speaks:

// Give a custom "sidebar" role both an inline and a block form:
const options = {
  format: "plain",
  contextualize: ["sidebar"],
  contextualization: {
    contextualizations: {
      sidebar: { inline: "Sidebar.", block: { start: "Start of the sidebar.", end: "End of the sidebar." } },
    },
  },
};

await extractUtterances(gnd, options);
// "Start of the sidebar." ... content ... "End of the sidebar." (block is the default when both forms exist)

// No verbosity level here — this is a single call, forced to "inline" outright.
// A shape that varies by verbosity level is a navigator concept, not this
// function's: see contextualizationOverrides.shapes in Playback.md.
await extractUtterances(gnd, { ...options, contextualization: { ...options.contextualization, shapes: { sidebar: "inline" } } });
// "Sidebar." ... content.

`contextualization.params`

Extra {{ placeholder }} values for a role, computed from its GuidedNavigationObject fields — for a placeholder the extractor has no built-in source for at all.

table/row/cell/rowheader’s own lines/columns/count/header/value don’t need this: reference them directly in your contextualizations wording and they resolve on their own, since the extractor always computes table structure regardless of whether this field is set.

// No params at all — {{ lines }}/{{ columns }} just work.
await extractUtterances(gnd, {
  format: "plain",
  contextualize: ["table"],
  contextualization: {
    shapes: { table: "inline" },
    contextualizations: { table: { inline: "Custom table, {{ lines }} by {{ columns }}." } },
  },
});
// → "Custom table, 3 lines by 2 columns."

params is for a role/placeholder the extractor has no built-in source for — e.g. list has no built-in item count the way table has lines/columns:

await extractUtterances(gnd, {
  format: "plain",
  contextualize: ["list"],
  contextualization: {
    contextualizations: { list: { block: { start: "List, {{ count }} items." } } },
    params: (role, node) => (role === "list" ? { count: node.children?.length ?? 0 } : undefined),
  },
});

`language`

How a node’s own inline spans (<em lang="fr">) render. Never merges across sibling utterances — each already has its own utterance and keeps it regardless.

"block-level" (default) — inline spans merge untagged into the surrounding text; block-level language kept.
"always" — ssml keeps spans tagged; plain splits into one utterance per language run.
"none" — same merge as "block-level", and every utterance’s language is contentLanguage, or none without it.

`contentLanguage`

The language of the content as a whole: content without lang/xml:lang is in it, and with language: "none", all content is. Sentences are segmented in it, with its suppressions, and languageOverrides applies to it. Without it, that content has no language, and sentence segmentation falls back to English. loadGndContent()’s { language } is passed as this option.

// <p>Es war einmal.</p><p lang="fr">Il était une fois.</p>
await extractUtterances(gnd, { format: "plain", contentLanguage: "de" });
// [{ language: "de", plain: "Es war einmal." }, { language: "fr", plain: "Il était une fois." }]
await extractUtterances(gnd, { format: "plain", language: "none", contentLanguage: "de" });
// [{ language: "de", plain: "Es war einmal." }, { language: "de", plain: "Il était une fois." }]

`languageOverrides`

The language content declared in a given language is spoken in, for content whose lang/xml:lang is wrong. Keys match like segmentation.suppressions: by primary subtag, an exact tag winning over it. Content declared in any other language is left as is, and an override applies before sentence segmentation, so segmenter and suppressions follow the spoken language.

// <html lang="en"><p>Bonjour <span lang="la">carpe diem</span>.</p></html>
await extractUtterances(gnd, { format: "plain", language: "always", languageOverrides: { en: "fr" } });
// [{ language: "fr", plain: "Bonjour" }, { language: "la", plain: "carpe diem." }]

`interruptions`

How each role in interruptionRoles (abbreviation, biblioref, glossref, noteref, pagebreak) is read where it interrupts the text: "never", "inline" at that exact point, or "after" (the default) — after the sentence holding it with segmentation: { mode: "sentence" }, after its whole block otherwise. Independent of skip and of verbosity presets.

noteref: its footnote. "never" drops it.
pagebreak: its label. "never" drops it.
biblioref/glossref: the bibliography entry or definition they embed; their own text is always read in place. With "after", a definition is read as “term: definition” (catalog’s glossref.term). With them in contextualize, “Bibliographic reference.” or “Glossary definition.” is read before the entry or definition.
abbreviation: its <sub alias> expansion, in parentheses right after the abbreviation with "after", in place of it with "inline"; "never" reads the abbreviation only.

// <p>The <abbr title="World Health Organization">WHO</abbr> said.</p>
await extractUtterances(gnd, { format: "plain" });
// [{ plain: "The WHO (World Health Organization) said." }]

await extractUtterances(gnd, { format: "plain", interruptions: { abbreviation: "inline" } });
// [{ plain: "The World Health Organization said." }]

// <p>...in the middle <span epub:type="pagebreak" title="Page 5"/> of a sentence.</p>
await extractUtterances(gnd, { format: "plain" });
// [{ language: "en", plain: "...in the middle of a sentence." }, { plain: "Page 5" }]

await extractUtterances(gnd, { format: "plain", interruptions: { pagebreak: "inline" } });
// [{ language: "en", plain: "...in the middle Page 5 of a sentence." }]

// <p>The light passes through a <a role="doc-glossref" href="#fresnel">Fresnel lens</a> at the top.</p>
// <dl role="doc-glossary"><dt id="fresnel"><dfn>Fresnel lens</dfn></dt><dd>A compact lens for focusing light.</dd></dl>
await extractUtterances(gnd, { format: "plain" });
// [{ plain: "The light passes through a Fresnel lens at the top." }, { plain: "Fresnel lens: A compact lens for focusing light." }, ...]

With segmentation: { mode: "sentence" }, an inline footnote or pagebreak stays inside the sentence holding it, its own sentences never ending that one. Between two sentences, it closes the one before it:

// <p>First with a note<a epub:type="noteref" href="#n1">1</a> here. Second sentence.</p>
// <aside epub:type="footnote" id="n1"><p>The note.</p></aside>
const options = { format: "plain", contextualize: ["footnote"], segmentation: { mode: "sentence" } };
await extractUtterances(gnd, { ...options, interruptions: { noteref: "inline" } });
// [{ plain: "First with a note Start of the footnote. The note. End of the footnote. here." }, { plain: "Second sentence." }]

await extractUtterances(gnd, options);
// [{ plain: "First with a note here." }, { plain: "Start of the footnote. The note. End of the footnote." }, { plain: "Second sentence." }]

A pagebreak opening a block is spoken before its first sentence.

`segmentation`

"structure" (default) — one utterance per structural/block-level unit, whatever its sentence count.
"sentence" — split at real sentence boundaries instead: a multi-sentence node becomes several utterances.

// <p>Hello there. This has two sentences.</p>
await extractUtterances(gnd, { format: "plain", segmentation: { mode: "sentence" } });
// [{ plain: "Hello there. " }, { plain: "This has two sentences." }]

Without reconstruct, a sentence only joins pieces of its own block, the nearest element with a block-level role (paragraph, listItem, cell, …), e.g. a <dfn> and the text after it. Unroled <div>s aren’t blocks, so their pieces join when a sentence runs across them. With reconstruct: true, a sentence genuinely split across blocks (e.g. a fixed-layout document with no enclosing paragraph, just positioned text fragments) is reconstructed into one utterance covering both:

// <p>This sentence continues</p><p>across two paragraphs.</p>
await extractUtterances(gnd, { format: "plain", segmentation: { mode: "sentence" } });
// [{ plain: "This sentence continues" }, { plain: "across two paragraphs." }]
await extractUtterances(gnd, { format: "plain", segmentation: { mode: "sentence", reconstruct: true } });
// [{ plain: "This sentence continues across two paragraphs." }]

Each utterance’s offsets (see above) says which source element(s) it was built from — up to one entry per contributing node, so a sentence reconstructed across two elements gets two entries, each with its own locate.

suppressions lists, per language, abbreviations (with trailing period, e.g. "d.") that shouldn’t be mistaken for sentence endings:

await extractUtterances(gnd, {
  format: "plain",
  segmentation: { mode: "sentence", suppressions: { en: ["approx."] } },
});

`segmenter`

The built-in sentence splitter is Intl.Segmenter with granularity: "sentence", plus this library’s own abbreviation-suppression pass on top (suppressions above, merged with builtInSuppressions). Risks worth knowing before relying on it:

ICU data availability. Intl.Segmenter’s sentence-break behavior depends on the JS engine’s bundled ICU/CLDR data. Runtimes that ship reduced ICU (some mobile WebViews, some server builds configured with small-icu) can produce different — sometimes worse — boundaries than a desktop browser, with no error raised.
Locale coverage is uneven. The Unicode sentence-break algorithm behind it is tuned mainly for widely-used languages; boundary quality for less common locales can be noticeably weaker, and there’s no per-locale opt-out — the same algorithm runs regardless of language.
No built-in abbreviation handling. Intl.Segmenter has no concept of “d.” or “Mr.” not ending a sentence; every such case is caught only by suppressions/builtInSuppressions, a fixed list rather than real disambiguation — an abbreviation missing from both lists still gets split on.

Supply segmenter to replace the built-in splitter entirely for "sentence" mode:

interface SentenceBoundary {
  text: string;
  start: number;
  end: number;       // reaches through trailing whitespace to the next sentence
  contentEnd: number; // start + text.length, no trailing separator — offsets use this, not `end`
}

type SentenceSegmenter = (
  language: string,
  text: string,
  customSuppressions?: string[]
) => Promise<SentenceBoundary[]>;

const mySegmenter: SentenceSegmenter = async (language, text, customSuppressions) => {
  // e.g. call out to a different sentence-boundary library
  return mySentenceLibrary.segment(text, { locale: language });
};

await extractUtterances(gnd, { format: "plain", segmentation: { mode: "sentence", segmenter: mySegmenter } });

A supplied segmenter fully replaces the built-in one — it’s handed the same per-language suppressions (as its own customSuppressions argument) but owns deciding what to do with them; builtInSuppressions is not merged in automatically, since that list exists specifically to patch Intl.Segmenter’s gaps.

Reconstruction heuristics

With reconstruct: true only. Input: the flat, ordered utterance list the walk (above) produced, plus each utterance’s source node’s ancestors in the GND tree.

Step 1 — eligibility. For each pair of adjacent utterances A, B, A may extend a run into B only if all of the following hold:

Both A and B have real source text (not missing/empty).
Neither A nor B was authored by extraction itself rather than lifted from the source — e.g. an image’s description (its alt text) is extraction-authored, since it stands in for text the document doesn’t have.
A and B have the same language, treating a missing language on either one as "en".
Neither A’s nor B’s source node carries a role where missing punctuation is not meaningful: cell, rowheader, row, table, list, listItem, heading1–heading6.
A and B sit in the same roled containers: no ancestor with a role other than presentation holds one but not the other. <p>A subtitle</p> inside a <header> never joins the <p> after it, and neither do two listItems. Unroled and presentation wrappers don’t count, so fixed-layout fragments under separate positioning <div>s still join.

A maximal run of pairwise-eligible utterances is built by scanning forward while eligibility holds.

Step 2 — confirmation. Take the whole run’s text, joined into one string, and run the real sentence segmenter on it once — segmentation is never decided any other way. For each gap between two pieces in the run:

The gap is a genuine join only if some detected sentence boundary extends past the gap into the next piece’s own text. A boundary that lands exactly at the gap, consuming none of the next piece’s text, does not count.
Consecutive genuine-join gaps chain into one merge group. That group is then resegmented on its own, self-contained text — a confirmed group is not guaranteed to collapse into exactly one utterance; it can still yield more than one.

Anything left outside a merge group is split on its own sentence boundaries independently.

`substitutions`

Rewrites ASCII-typed imitations of Unicode symbols in plain/ssml before an utterance is returned, so the imitation is spoken as the symbol it stands in for rather than read literally. Merged by key on top of builtInSubstitutions — a caller-supplied key replaces the built-in rule of the same name rather than adding to it.

// <p>See page 5, fig. 2.</p>
await extractUtterances(gnd, { format: "plain", substitutions: { "fig.": "figure" } });
// plain: "See page 5, figure 2."

A rule value is either a plain string, matched as a whole token (like "fig." above), or { pattern, replace } for a match that needs its own regex:

// <p>See p.5 for details.</p>
await extractUtterances(gnd, {
  format: "plain",
  substitutions: { "p.": { pattern: /p\.(?=\d)/g, replace: "page " } },
});
// plain: "See page 5 for details."

Unlike segmentation.suppressions, this table is flat, not per-language — the voice speaks the resulting symbol correctly regardless of the utterance’s own language.

Contextualization catalog

Each entry is keyed by GndRole and resolved through i18next, so wording is a translatable JSON resource, not code:

{
  "figure": { "inline": "Figure: {{ description }}" },
  "table": {
    "block": {
      "start": {
        "labelled": "Table: {{ description }}. {{ lines }}. {{ columns }}.",
        "unlabelled": "Table. {{ lines }}. {{ columns }}."
      },
      "end": "End of the table."
    },
    "inline": {
      "labelled": "Table: {{ description }}. {{ lines }}. {{ columns }}.",
      "unlabelled": "Table. {{ lines }}. {{ columns }}."
    },
    "parts": { "lines_one": "1 line", "lines_other": "{{ count }} lines", ... }
  }
}

`inline` vs `block`

inline — spoken once, before the node’s content.
block: { start, end } — start spoken before the content, end spoken after.

A role can have either one or both. table has both — see contextualization.shapes above for how that choice is made.

Variants

A catalog value can be a plain string, or a small object of named variants — labelled/unlabelled, withHeader/withoutHeader — that the extractor picks between based on the node:

audio/video/image/math/table (and any other role with a description) pick labelled when the node has one, unlabelled otherwise.
cell/rowheader pick withHeader when a column header was found for that cell, withoutHeader otherwise.

`parts`

Pluralizable fragments a role’s own wording references as tokens ({{ lines }}), using i18next’s plural-category key suffixes (lines_one, lines_other, …) so each locale supplies its own plural forms.

Built-in placeholders

Each placeholder below is filled in automatically — no contextualization.params needed:

{{ description }} — for any node with a description, whatever its role.
{{ lines }} / {{ columns }} — table’s own row/column counts.
{{ count }} — a row’s 1-based position.
{{ header }} / {{ value }} — a cell/rowheader’s column-header text and content. GND carries no colspan/rowspan, so a cell’s header is matched by position, not by explicit association.

Any other placeholder — one not in this list, or on a role of your own — needs a contextualization.params entry.

Implicit accessible names

A <table>/<figure> with no explicit ARIA name (aria-label/aria-labelledby) but a <caption>/<figcaption>/role="caption" child folds that child’s text into its own description, instead of speaking it as a separate node — matching HTML-AAM’s implicit accessible name computation.

Fixtures

Each fixture’s utterances.json is the hand-reviewed expected output for one option combo — see fixtures/README.md and Testing.

This site is open source. Improve this page.

speech

Utterance Extraction

Options

format

skip

contextualize

Contextualization

contextualizationLocale

contextualization.contextualizations

contextualization.shapes

contextualization.params

language

contentLanguage

languageOverrides

interruptions

segmentation

segmenter