Subtitles (speech to text)¶

Sapsan can recognize speech in a recorded stream and write it into a separate subtitle track. Recognition runs on the built-in Whisper engine (whisper.cpp) over the DVR archive: a background worker reads recorded audio, transcribes it and appends subtitle frames back into the archive. The subtitle track is then served in HLS and DASH as WebVTT.

Because the worker reads from the archive rather than tapping the live stream, it can never break the stream: if recognition cannot keep up, subtitles simply lag behind and catch up later. This is why DVR must be enabled for the stream.

Prerequisites¶

1. A build with the whisper feature. Speech recognition links a native whisper.cpp + ffmpeg stack, so it is off by default:

   bash make release FEATURES=whisper # or: cargo build --release --features whisper

   On Apple Silicon add Metal for GPU acceleration (much faster): --features whisper,metal.

2. DVR enabled on the stream (the worker reads and writes the archive).

3. A model file. Models are named by a short name (tiny, base, small, medium, large-v3); Sapsan resolves the name to a GGML file ggml-<name>.bin in the models directory:

   • WHISPER_MODELS_DIR if set, otherwise ~/.cache/whisper-models.

   Download a model once, for example medium:

   bash mkdir -p ~/.cache/whisper-models curl -L -o ~/.cache/whisper-models/ggml-medium.bin \ https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.bin

Configuration¶

Add a speech2text block to a stream that already has DVR:

streams:
  news:
    inputs:
    - rtsp:
        url: rtsp://admin:password@10.0.0.5/stream0
    dvr:
      root: /storage
    speech2text:
      model: medium        # tiny | base | small | medium (default) | large-v3
      language: ru         # ISO-639-1; omit for auto-detection
      audio_track: a1       # source audio track (default: first audio track)
      max_window_secs: 30   # recognition window cap, s (default 30, max 120)

Fields:

Validation: a stream with speech2text but no dvr is rejected; max_window_secs must be in 1..=120; language must be a 2–3 letter code.

Changes apply on reload (SIGHUP / central): the worker is restarted only if the speech2text block actually changed.

Verifying offline (before enabling on a stream)¶

The subtitle CLI (built with the whisper feature) runs the exact production recognizer on a local file, writes the result next to it and, if the file already carries a subtitle track, reports a closeness metric. This is the fastest way to pick a model and language and to check quality:

subtitle recognize movie.mkv --lang ru --model medium
# process only a segment (e.g. 2 minutes from 25:00) for a quick check:
subtitle recognize movie.mkv --lang ru --start-sec 1500 --limit-sec 120

Outputs, next to the input file:

• movie.subtitles.json — structural subtitle frames with full provenance (audio window, model, hash, prompt, per-cue diagnostics), enough to reproduce a run;
• movie.vtt — the WebVTT projection;
• movie.diff.txt — a side-by-side of reference vs recognized text (when the file has a reference subtitle track).

If the file has reference subtitles, the CLI prints WER (word error rate) and CER (character error rate), globally and per window. Note: against an independently-authored subtitle track WER measures translation divergence, not ASR error — read it as a relative signal between runs and eyeball the diff.

Any .mkv/.mp4 file works as input.

Verifying on a stream¶

Once speech2text is enabled on a DVR stream:

1. The stream's media info gains a subtitle track. Check the HLS master playlist — it now contains an EXT-X-MEDIA:TYPE=SUBTITLES rendition and the variants reference it with SUBTITLES="subs":

   bash curl -s "http://server/streaming/<stream>/index.m3u8" | grep -i subtitle

   In DASH the manifest gains a contentType="text" AdaptationSet.

2. Open the archive in a player that supports subtitles (or the HLS/DASH URL) and enable the subtitle track. Subtitles appear with a lag while the worker catches up to the live edge, then track it.

3. Watch the recognition counters in the metrics endpoint (processed seconds, gap size / lag, errors).

Because the worker is stateless — progress is derived from the edge of the subtitle track in the archive — it resumes from the same gap after a restart, backfills a stream that already has an archive, and never duplicates a frame.

How it works¶

The worker's unit of work is the gap between audio coverage and subtitle coverage in the archive. It reads the audio of the gap, decodes it to 16 kHz mono PCM, runs Whisper over sliding windows (with a silence gate against hallucinations and re-anchoring on complete segments), and appends structural subtitle frames. WebVTT delivered to players is a projection of those frames; the stored frames additionally keep provenance and Whisper diagnostics for debugging.

Next steps¶

• Configure DVR recording
• HLS playback