Readium Logo

Streamer API

Summary

This proposal aims to specify the Streamer public API and showcase how a reading app might support additional formats. It ties together several concepts introduced in other proposals such as the Composite Fetcher API, Publication Encapsulation and the Publication Helpers & Services.

Motivation

We’re trying to address the following needs and pain points for reading apps.

Supported Formats

A reading app might want to:

Publication Extensibility

A reading app should be able to customize which Publication Services are added to a Publication object by adding, removing or replacing services.

It should also be able to replace or decorate the root Fetcher, for example to add resource transformers or handle resources caching.

Replacing Third-Party Dependencies

On some platforms, Readium needs to import third-party dependencies, for example to parse PDF or XML documents. When possible, these dependencies should not be hard-coded in the toolkit, because:

Instead, generic interfaces should be declared in r2-shared, with a default implementation using the third-party dependencies chosen by Readium.

Using interfaces allows to write a single unit test suite shared between all third-party implementations, which is useful to perform benchmarking comparisons or ensure that migrating to a different library won’t break the toolkit. Here’s an example comparing Minizip and ZIPFoundation for Swift.

Typically, interfaces might be useful for PDF, XML, HTTP and archiving (ZIP, RAR, etc.) libraries.

Customizing Parsers

While Readium ships with sane default parser settings, some degree of configuration might be offered to reading apps. These settings are format-specific and thus can’t live in the Streamer API itself, but the Streamer should allow reading apps to set these settings one way or the other.

Developer Guide

The Streamer is one of the main components of the Readium Architecture, whose responsibilities are to:

Usage

Opening a Publication Asset

Opening a Publication is really simple with an instance of Streamer.

asset = FileAsset(path)
streamer = Streamer()
publication = streamer.open(asset)

Your app will automatically support parsing new formats added in Readium. However, if you wish to limit the supported formats to a subset of what Readium offers, simply guard the call to open() by checking the value of asset.mediaType first.

supportedMediaTypes = [MediaType.EPUB, MediaType.PDF]
if (!supportedMediaTypes.contains(asset.mediaType)) {
    return
}

Alternatively, you can provide the parsers yourself and disable the default ones with ignoresDefaultParsers.

streamer = Streamer(
    parsers = [EPUBParser(), PDFParser()],
    ignoresDefaultParsers = true
)

Customizing Parsers

If a parser offers settings that you wish to override, you can create an instance yourself. It will automatically take precedence over the defaut parser provided by the Streamer.

streamer = Streamer(
    parsers = [EPUBParser(...)]
)

Customizing the Parsed Publication

You can customize the parsed Publication object by modifying:

The Streamer accepts a Publication.Builder.Transform closure which will be called just before creating the Publication object.

streamer = Streamer(
    onCreatePublication = { mediaType, manifest, fetcher, services ->
        // Minifies the HTML resources in an EPUB.
        if (mediaType == MediaType.EPUB) {
            fetcher = TransformingFetcher(fetcher, minifyHTML)
        }

        // Decorates the default PositionsService to cache its result in a
        // persistent storage, to improve performances.
        services.decorate(PositionsService::class) { oldFactory ->
            CachedPositionsService.createFactory(oldFactory)
        }

        // Sets a custom SearchService implementation for PDF.
        is (mediaType == MediaType.PDF) {
            services.searchServiceFactory = PDFSearchService.create
        }
    }
)

Providing Different Implementations of Third–Party Services

The Streamer and its parsers depend on core features which might not be available natively on the platform, such as reading ZIP archives or parsing XML. In which case, Readium uses third-party libraries. To use a different version of a library than the one provided by Readium, or use a different library altogether, you can provide your own implementation to the Streamer.

streamer = Streamer(
    httpClient = CustomHTTPClient(),
    pdfFactory = { path, password -> CustomPDFDocument(path, password) }
)

If you just want to add HTTP headers or set up caching and networking policies for HTTP requests, you can instantiate the default HTTP client yourself. No need to create a custom implementation of HTTPClient.

Supporting Custom Formats

The Readium Architecture is opened to support additional publication formats.

  1. Register your new format and add a sniffer. This step is optional but recommended to make your format a first-class citizen in the toolkit.
  2. Implement a PublicationParser to parse the publication format into a Publication object. Then, provide an instance to the Streamer.
class CustomParser: PublicationParser {

    func parse(asset: PublicationAsset, fetcher: Fetcher, warnings: WarningLogger?) -> Publication.Builder? {
        if (asset.mediaType != MediaType.MyCustomFormat) {
            return null
        }

        return Publication.Builder(
            manifest = parseManifest(from: asset),
            fetcher = fetcher
            services = [CustomPositionsServiceFactory()]
        )
    }

    private func parseManifest(from asset: PublicationAsset) -> Manifest {
        ...
    }

}

streamer = Streamer(
    parsers = [CustomParser()]
)

Backward Compatibility and Migration

Mobile (Swift & Kotlin)

Reading apps will still be able to use indivual parsers directly. However, we should strongly recommend them to migrate to this new way of opening a Publication, which will simplify their integration and make sure they automatically benefit from new formats supported by Readium.

Reference Guide (r2-shared)

PublicationAsset Interface

Represents a digital medium (e.g. a file) offering access to a publication.

Properties

Methods

FileAsset Class (implements PublicationAsset)

Represents a publication stored as a file on the local file system.

Constructors

Properties

Publication.OpeningError Enum

Errors occurring while opening a Publication.

WarningLogger Interface

Interface to be implemented by third-party apps if they want to observe non-fatal warnings raised, for example, during the parsing of a Publication.

Methods

Warning Interface

Represents a non-fatal warning message which can be raised by a Readium library.

For example, while parsing a publication we might want to report authoring issues without failing the whole parsing.

Properties
Warning.SeverityLevel Enum

Indicates how the user experience might be affected by a warning.

ListWarningLogger Class (implements WarningLogger)

Implementation of WarningLogger which accumulates the warnings in a list, to be used as a convenience by third-party apps.

Reference Guide (r2-streamer)

Publication.Builder Class

Builds a Publication from its components.

A Publication’s construction is distributed over the Streamer and its parsers, so a builder is useful to pass the parts around before actually building it.

Constructors

Methods

Publication.Builder.Transform Function Type

typealias Publication.Builder.Transform = (
    mediaType: MediaType,
    manifest: *Manifest,
    fetcher: *Fetcher,
    services: *Publication.ServicesBuilder
) -> Void

Transform which can be used to modify a Publication’s components before building it. For example, to add Publication Services or decorate the root Fetcher.

The signature depends on the capabilities of the platform: manifest, fetcher and services should be modifiable “in place”, hence the pseudo-pointers types.

PublicationParser Interface

Parses a Publication from an asset.

Methods

Streamer Class

Opens a Publication using a list of parsers.

Constructor

The specification of HTTPClient, Archive, XMLDocument and PDFDocument is out of scope for this proposal.

Methods

PublicationParser Implementations

These default parser implementations are provided by the Streamer out of the box. The following is not meant to be a full parsing specification for each format, only a set of guidelines.

ReadiumWebPubParser Class

Parses a Publication from a Readium Web Publication or one of its profiles: Audiobook, DiViNa and LCPDF.

Both packages and manifests are supported by this parser.

W3CWPUBParser Class

Parses a Publication from a W3C Web Publication or one of its profiles, e.g. Audiobook.

The W3C to RWPM mapping is documented here.

EPUBParser Class

Parses a Publication from an EPUB publication.

The EPUB parser is already extensively documented.

PDFParser Class

Parses a Publication from a PDF document.

Reference: PDF 1.7 specification

Reading Order

The reading order contains a single link pointing to the PDF document, with the HREF /<asset.name>.

Table of Contents

The Document Outline (i.e. section 12.3.3) can be used to create a table of contents. The HREF of each link should use a page= fragment identifier, following this template: /<asset.name>#page=<pageNumber>, where pageNumber starts from 1.

Cover

The cover should be generated by rendering the first page of the PDF document.

Metadata

(See section “14.3 Metadata” of the PDF 1.7 specification)

Metadata can be stored in a PDF document either with a metadata stream (1.4+) or with a document info dictionary. The metadata stream is the preferred method and therefore takes precedence, but the parser should be able to fallback on the document info dictionary.

Identifier

The Publication’s identifier should be computed from the PDF’s file identifier (i.e. section 14.4), located in the trailer dictionary.

/ID[<491b7e3d57fa8ca81da62895cbdb22fe><1317e207f71ec2dcff49a219b869606d>]

It’s a pair of two identifiers, the first one being permanent while the second one is generated for every change in the PDF document. It could be useful to preserve both identifiers, so the Publication’s identifier should be computed as <id-created>;<id-modified>.

ImageParser Class

Parses an image–based Publication from an unstructured archive format containing bitmap files, such as CBZ or a simple ZIP. It can also work for a standalone bitmap file.

To be recognized by this parser, any of these conditions must be satisfied:

Reading Order

The reading order is built by sorting the fetcher’s links by their HREF. Only resources recognized as bitmap files are added to the reading order.

Table of Contents

If the links contain intermediate folders, their names are used to build a table of contents. However, folders which don’t contain any bitmap descendants are ignored. Each table of content item points to the first bitmap resource listed in the folder.

Metadata

There’s no standard way to embed metadata in a CBZ, but two formats seem to be used in the wild: ComicRack and ComicBookInfo. EmbedComicMetadata is a plugin for Calibre handling different CBZ metadata formats.

Following common practice, if the archive contains a single root folder containing the bitmaps, the publication title is derived from it.

More information at MobileRead.

AudioParser Class

Parses an audiobook Publication from an unstructured archive format containing audio files, such as ZAB (Zipped Audio Book) or a simple ZIP. It can also work for a standalone audio file.

To be recognized by this parser, any of these conditions must be satisfied:

Reading Order

The reading order is built by sorting the fetcher’s links by their HREF. Only resources recognized as audio files are added to the reading order.

Table of Contents

If the links contain intermediate folders, their names are used to build a table of contents. However, folders which don’t contain any audio clip descendants are ignored. Each table of content item points to the first audio resource listed in the folder.

Metadata

There’s no standard way to embed metadata in a ZAB, but there are a number of playlist formats which could be used. M3U seems to be the most popular. Individual audio format metadata could also be used, in particular for the reading order titles.

Following common practice, if the archive contains a single root folder containing the audio files, the publication title is derived from it.

Rationale and Alternatives

By adding the Streamer object, we aimed to provide an API that is simple to use while still allowing some flexibility.

An alternative that is currently in use in the mobile toolkits would be to provide a set of parsers and leave the responsibility to select and call the parser matching the publication file to reading apps. However, this strategy has downsides:

Drawbacks and Limitations

Some parsers, such as PDFParser, might depend on heavy libraries which are directly linked into the toolkit. This situation is problematic for applications which are not interested in these formats, or would like to replace the dependency with another one, because it increases significantly the size of the app.

We might want to offer sub-libraries for such heavy parsers, to keep the toolkit lightweight.

Future Possibilities

Handling content protection technologies is a complex subject, which deserves its own proposal. Because the Streamer is the gateway to parsers and fetchers, it’s a place of choice to add support for content protections.

The Streamer is also a good place to handle injection in publication resources, such as JavaScript/CSS injection in HTML resources. Promoting the concept of “injectable” as a first-class type to plug in the Streamer (and/or the Navigator) would be very useful for reading apps.

This site is open source. Improve this page.