Subtitle OCR¶

media.process.ocr_subtitles

Sapsan converts DVB bitmap subtitles from an incoming stream into separate text tracks. The broadcaster sends an image of a subtitle cue; OCR recognizes the characters and associates the resulting text with its display time. Viewers get subtitles they can select in the player menu, even if the player cannot display the original DVB images.

Processing runs on the ingest streamer while the live stream is running. Archive recording is not required to run OCR. If DVR is enabled, the generated text track can be recorded alongside the stream.

Which subtitles are recognized¶

The input is a separate EN 300 743 DVB graphical subtitle service. In MPEG-TS, its description is carried in the PMT: this tells Sapsan the language and page identifiers. Recognition processes the subtitle bitmap, not the entire video image.

Different text sources have their own processing methods:

• DVB bitmap subtitles — images containing text; the subtitles section and the OCR described here are intended for them.
• Teletext — already contains character codes; extraction is configured in the teletext section.
• CEA-608/708 — closed captions inside video, controlled by the closed_captions section.
• Speech in an audio track — a separate speech recognition feature.

This OCR pipeline does not extract text already burned into the video image. Recognized text keeps the source service's language; automatic translation into another language is not performed here.

How an image becomes a text track¶

1. Service discovery. Sapsan reads the input track descriptions and discovers DVB subtitle languages. You do not need to list PIDs and pages manually in the configuration.
2. Image assembly. The DVB decoder reconstructs a page from the transmitted objects and palettes. Regions of one composition are assembled into an image for recognition.
3. Recognition. The image is passed to the selected engine. Retransmitting an unchanged composition does not require another OCR pass.
4. Cue preparation. Empty lines and some recognition artifacts are removed from the result. The text receives display timing on the stream timeline. Changing or clearing the page ends the previous cue.
5. Delivery. Cues enter the shared text track processing pipeline and are converted to WebVTT for delivery. Playback is available through HLS and DASH.

A background worker performs recognition. Its work queue is bounded: under overload, some jobs may be dropped and their text lost. When assessing performance, monitor subtitle delay and the queue drop counter as well as video.

Text is delivered in windows tied to segment duration. A recognized cue can be delivered in successive windows while it remains on screen: its first delivery does not have to wait until it disappears. Empty intervals are also accounted for, so the subtitle track continues advancing through pauses.

Latency depends on OCR and segmentation time, as well as player buffering. When a text track appears, Sapsan disables LL-HLS for the entire stream: HLS delivery switches to regular segments without LL-HLS parts. Account for this when choosing a mode for low latency broadcasting. Check when text appears using the actual output pipeline.

Recognition engines¶

Sapsan supports two engines.

Tesseract recognizes images using a language model. A compatible Tesseract library and language model traineddata files for the input stream's languages must be available on the streamer. Result quality depends on the font and source bitmap; having a model does not guarantee error-free text.

glyphtable matches character images against a labeled glyph table. The table is embedded in Sapsan; this engine does not require a separate OCR library. This method suits broadcasters using known glyph shapes represented in the table. Unknown glyphs become replacement characters; a new typeface or rendering method requires checking the table's coverage.

If engine is omitted, Sapsan first tries to initialize Tesseract for the discovered language, then glyphtable if Tesseract initialization fails. This selection happens when the worker starts. Poor recognition quality for a cue does not itself trigger another recognition pass with a different engine.

With an explicit engine: tesseract or engine: glyphtable, only the selected engine is used. If it is unavailable, Sapsan shows the reason in statistics and does not substitute another engine.

Configuration¶

Configuration is supplied through the central API. To enable OCR, add a subtitles section to the stream:

streams:
  - name: tv1
    inputs:
      - udp:
          host: 239.0.0.1
          port: 5000
    subtitles:
      mode: extract

Here, Sapsan discovers languages in the input stream and automatically selects an available engine. To explicitly select the glyph table, use this section:

subtitles:
  mode: extract
  engine: glyphtable

Three modes are available:

• passthrough — the default mode. Original bitmap subtitles are retained in MPEG-TS, OCR does not run, and no text tracks are created.
• extract — recognition creates text tracks while retaining the original bitmap subtitles. This mode lets you evaluate OCR while continuing to serve devices with a DVB decoder.
• replace — recognition creates text tracks and removes the original subtitle PID bound to a working worker from MPEG-TS output.

In replace mode, the entire PID is removed, including other language services on it. If some languages could not be recognized, their bitmap subtitles may also disappear from output. Check every language before choosing this mode.

If the engine fails to start, removal of its source PID is not enabled. If service binding is lost or the worker fails permanently, the source PID returns, unless another working worker on the same PID still removes it. An individual OCR error or queue overflow does not necessarily stop the worker, so automatic bitmap restoration for every missed cue is not guaranteed.

Resources on the ingest streamer¶

The Tesseract library and language data must be accessible to the Sapsan process on the server ingesting the stream. Installing these resources only on central does not enable recognition.

For resources in custom locations, use process environment variables:

A custom table completely replaces the embedded table, rather than extending it. If the file is unavailable or lacks the required language, the embedded table is not substituted automatically.

After installing or replacing the Tesseract library and changing the environment, restart the streamer process: the library loading result is cached. Adding model files also does not mean that an already unavailable worker will automatically try loading them again; after resolving the cause, recreate the worker by changing the subtitles section itself or restarting the streamer.

Languages and source changes¶

Sapsan creates one OCR track per discovered language, up to nine tracks per stream. The language comes from the DVB service description. If the broadcaster signals the wrong language, this affects recognition data selection and the output track label.

If several services use the same language, the first in PMT order is selected. Regular subtitles and a hearing-impaired variant of the same language do not become two independent OCR tracks.

When the input description changes, Sapsan reassesses service bindings. If a language temporarily disappears, its output track is retained and the worker enters no_source_match. Retaining tracks preserves their identity for consumers, but means the nine-track limit counts tracks already created, not just languages currently present.

When configuring source failover, check that DVB services are present and correctly labeled on every input. Successful video switching alone does not confirm that subtitles have been preserved.

Monitoring¶

OCR state is available in the subtitles array of stream statistics on the ingest streamer:

curl -fsS -u "$ADMIN_USER:$ADMIN_PASSWORD" \
  http://streamer:8080/streamer/api-v4/streams/stats/tv1 \
  | jq '{name, subtitles: [(.subtitles // [])[] | {
      track, producer, lang, state,
      source_pid, source_page_id, source_lang,
      coverage_lag_ms, error
    }]}'

The source_pid and source_page_id fields identify the DVB service bound to the worker. coverage_lag_ms is the lag, in milliseconds, of the processed subtitle interval's edge behind the stream timeline. This field may be absent before the first output; a missing value does not mean zero delay.

Main states:

• running — the worker is running and bound to a source. This does not assess recognition accuracy.
• no_source_match — the worker exists, but the matching service is currently absent. Binding is reassessed when the input description changes.
• no_model — language data is unavailable, or automatic selection could not start either engine. Details are in error.
• feature_missing — the selected engine is unavailable. For example, the Tesseract library may not be installed or may have failed to load.
• failed — the background worker has stopped because of a failure. It must be recreated after the cause is resolved.

An empty subtitles array alone does not prove that a model is missing: with extract enabled, it can also occur before the first DVB service is discovered.

Processing errors can be monitored through /streamer/api-v4/live-metrics metrics, if metric export is available in your distribution:

Error counters are cumulative: compare their change over an interval to assess current operation. They are not included in the subtitles array returned by the API request above.

Verifying the result¶

1. Enable extract and confirm that the input stream actually contains DVB bitmap subtitles.
2. Check the discovered languages and PID/page bindings. Confirm that workers have entered running.
3. Select the text track in the player. Compare cues against the original image across different programmes; check characters, line breaks, and synchronization with video.
4. Check delay and increases in error counters under production load. Separately assess replacement characters when using glyphtable.
5. If you need replace, check its effects on every language on the source PID. If you need an archive, confirm that a recorded fragment plays back with the text track.

DVB object bitmap coding with coding_method = 0 is supported; this decoder does not handle other object coding methods. Corrupt input data and insufficient processing capacity can cause missing cues. Checking the running state must be complemented by checking the text itself.