Skip to content

Streamer autonomy

Central is the control plane, not the media path. Everything a channel does for the viewer, the streamer does itself: ingests the source, transcodes, writes the archive, serves the stream. A central outage is therefore a management incident, not an on-air one. This page covers what exactly keeps working, what freezes, and how the cluster reassembles; everything described has been verified on a live cluster.

What keeps working

  • The air. Channels on the streamers keep broadcasting: everything a channel does on its machine — sources, the transcoder, archive recording, pushes, serving — does not depend on central. Viewers already directed to a streamer keep watching.
  • DVR recording. The archive is written without pauses — the window keeps growing throughout central's downtime.
  • Even a streamer restart. A streamer restarted while central is unreachable brings its channels up immediately — a warm start from the local cache of the configuration slice, without waiting for a single answer from central. It simply keeps retrying the sync:
WARN  sync failed, retrying   error sending request for url (.../node-api-v4/sync)

What freezes

  • Management. The console, the API, the channel list, statistics, sessions — the whole central plane. Configuration changes are not delivered, the layouter moves nothing: the cluster is frozen in its last placement.
  • The viewer path through central. Media proxying and redirect issuing go through central — new viewers coming to its address get no answer. Hence the production rule: give the streamers a public payload URL — the viewer's dependence on central then shrinks to a single redirect, and those already connected are not touched by the outage at all.
  • The local streamer. The streamer inside the central process shares its fate: channels placed on it fall together with the process. One more argument for keeping central and broadcasting on separate machines in production.

How the cluster reassembles

The streamer retries the sync with an exponential pause (capped at about a minute), so when central returns, the cluster knits itself back within that pause: in the registry the streamer walks from "Signal lost" back to "Online", statistics and sessions come alive. Nothing needs restarting.

Two special cases of central's return:

  • Central restored from a backup, with configuration versions older than what the streamer has applied. The streamer gets an explicit signal from central, takes the full snapshot as the truth and keeps working off it. A version rollback does not tear the cluster.
  • Central returned with an empty database — the disk is lost, there is no backup. The streamer is a stranger to it now, and its log says so outright:
ERROR node key rejected; streams keep running from the applied slice, operator action required

The air keeps running — the streamer lives off the applied slice. To bring it back into the cluster: stop the streamer service on the machine, clear its state directory (/var/lib/catena — the streamer's identity lives there; do not touch the archive if it sits on its own disk), open the join window and join the machine with a fresh token. The streamer arrives as a new registry record, and the layouter lays the channels out again. A token without the state reset does not help: the stored identity is stronger.

What pays for it

The resilience is not accidental — it is built in:

  • All communication is initiated by the streamer. Central has no reverse control channel: the streamer needs central to change the configuration, not to work.
  • The applied slice is stored on the streamer. The local configuration cache is what gives the warm start without central and life under the last placement for however long it takes.
  • Central has a single carrier of state — PostgreSQL. It has no state directory of its own: if the database backup is intact, the whole installation is intact. A regular pg_dump is the only backup the control plane needs.