Skip to content

Speech recognition

The station does not invent teletext and captions: it carries the text the picture brought with it. A channel that has no text at all gets no subtitles out of that — and it is exactly the channel that most often needs them: a regional insert, a corporate broadcast, a third-party source without service data.

Speech recognition covers that case: the station reads the audio of the recorded channel itself and appends a subtitle track to it.

The live signal is not touched at all. Recognition works on top of the archive: it reads the recorded audio and writes the subtitles back into it. Two consequences are worth knowing up front.

  • The channel must have archive recording. Without it recognition has no audio to take and nowhere to write the result.
  • Falling behind is safe. If recognition does not keep up with the live edge, subtitles lag and then catch up; the channel's output is not affected in any way.

Turning it on

Recognition lives on the Processing tab of the channel card, in the Speech recognition section. The Recognise speech into subtitles switch turns it on.

The speech recognition section on the processing tab

There are four fields and none of them is required — the defaults are enough to turn recognition on with a single switch.

Field What it sets
Model The size of the recogniser. A larger model is more accurate, but slower and needs more memory.
Language An ISO-639-1 code, for example en. Empty means detect from the audio.
Audio source The track of the feed to listen to. The first one by default.
Recognition window, s Up to 120 seconds. A longer window reads meaning better, but a cue appears later.

Set the language when you know it. Detection decides on the first seconds of audio and gets it wrong on a channel that opens with a musical ident — and a wrong language spoils the whole track, not a single cue.

Audio source is everyday business on a headend: a channel carries several language tracks, and recognition listens to exactly one. If the chosen track is not on air, the state says so — "The selected audio track is not on air".

It does not start without recording

Turn recognition on for a channel without recording and the panel answers at once, without waiting for a save:

The warning that recognition needs archive recording

This does not forbid saving the setting; it warns that the setting will not work. Give the channel recording and recognition comes up on its own.

Recognition state

Below the form the panel shows what is actually happening. This is the one place where the live work of recognition is visible: the channel card takes the state from the node where the channel is recorded.

Recognition state: running, lag, audio recognised, cues

  • Running / Not running — whether recognition is up. "Not running" always comes with a reason in words.
  • Lag — how far the recognised part is behind the present moment. At the live edge this is seconds; on an archive that recognition is catching up with from the beginning it is hours, and that is a normal working mode, not a failure.
  • Audio recognised — how much audio has already gone through the recogniser.
  • Cues — how many subtitle lines have been written.
  • Errors — archive read and recognition failures. Zero here is the usual state.
  • Last cue — when recognition last wrote anything.

The panel names the reasons recognition does not come up directly:

What it says What to do
The stream has no archive recording Turn recording on.
The model file is missing or failed to load Put the model file on the server (below).
The selected audio track is not on air Choose another audio source or leave the default.
The recogniser does not know this language Check the language code or clear the field.
This server build has no speech recognition Contact support: the box is built without the engine.

Recognition that runs and has a recent error is a warning, not an outage: one unread chunk does not stop the work, recognition carries on from the next one.

The model file

The model itself does not ship with the server: a usable one weighs over a gigabyte, and which one is wanted is the station owner's choice. The server looks for it in the product's state directory — /var/lib/mcaster/whisper-models — and expects a ggml-<model>.bin file there, that is ggml-medium.bin for the medium model.

While the file is missing, the recognition state honestly says "The model file is missing or failed to load", and the channel keeps running and recording as if nothing happened.

One file serves every channel of the station: the model is read into memory once per process, however many channels ask for it.

What goes out

What has been recognised becomes an ordinary subtitle track of the channel: it shows up in the media info and travels into the OTT copy as WebVTT — exactly where a text track is needed, because no OTT player decodes teletext.

The transport stream will not carry the recognised text: a subtitle track has nothing to travel in inside MPEG-TS — the same limitation as teletext extraction.

It makes sense to watch the track on the archive. At the live edge the subtitles are not there yet — they appear with exactly the lag the state shows.

What next