Redundancy¶

Streamer autonomy covers the air: a central that goes down is nothing to a viewer, streams keep broadcasting and recording on their own. Redundancy covers the second half — the control plane: so that a central outage does not stretch out over however long it takes to bring the machine back by hand.

This page is about what an installation with a spare is made of, how many instances of each component to run, and what exactly the second one changes. How to deploy it — A central group on separate servers; how to see it with your own eyes — trying it in Docker. The map of state and guarantees — Components and reliability.

What is made redundant and what is not¶

Only the control plane needs a spare. The media path does not depend on it: the streamer takes its source, transcodes, records and serves the stream itself, and a viewer given a public delivery address does not go through central at all.

Hence the rule any installation with a spare starts from: central and broadcasting live on different machines. A local streamer inside the central process will not do for such an installation — it shares the fate of the process, and its streams die together with the control plane.

The mandate: which instance does the work¶

Several central instances over one database make a group. Part of the work in it is harmless to do everywhere at once, and part is not: two layouters deciding at the same time would start moving streams between machines for no reason.

Work like that is done by one instance in the group — the one that has taken the mandate for it. Three properties of a mandate determine everything about how the group behaves under failure.

• A mandate is issued by PostgreSQL, not by the config. The config of an instance carries only a claim — "this machine is ready to do this work". It has to be declared everywhere, or there will be nobody to pick the work up.
• A mandate is released by the death of the process, by itself. It lives on a connection of its own to the database: the session is gone, the mandate is free, and a neighbour comes for it on its own. No timeouts, no sorting out of stuck executors, no manual confirmation.
• Every issue carries a number — an epoch. It grows when the holder changes and travels with every write. An instance that wakes up from a pause no longer the holder is refused at the write instead of corrupting the placement after the fact.

There are several mandates and they are independent. Central has no single leader: the placement and the picture of what is happening right now may be held by different machines, and that is a normal state.

Who holds what is visible in the console: Cluster → Central.

Components and their count¶

Central is not one process with one role but several different jobs, and they are made redundant in different ways.

What follows is what each of them does and why the count is what it is.

Request handlers¶

This is what the operator and the cluster see as central: the console, the management API, the player front end, the metrics export and the incoming sync from streamers. A handler holds no state of its own — everything lives in PostgreSQL — so there may be any number of instances, and any of them can answer an operator's request.

A streamer, however, needs not a balancer smearing it across the group but a list of addresses: it sticks to whichever address answers and moves on to the next after a failed request. The list is set in the streamer's config and does not extend itself — having added a third instance, write its address into the streamers.

The second instance gives two things:

• The control plane survives the loss of a machine. The operator and the streamers move to a live instance.
• Upgrades without a window of downtime. Instances are upgraded one at a time — see Upgrade.

The current picture¶

What the console shows about "right now" — which streams are running, who is watching them, how many viewers — central does not read from the database. The database holds what should be (settings, placement) and what has already happened (closed sessions, journals), while "right now" lives in memory.

The picture is one per group, and it is held by the holder of the stats mandate. The other instances reach it over the network, at the address the holder advertised for itself. This is why the whole park is visible at any address of the group even though streamers report to different instances.

Hence the one thing an operator needs to know about a change of holder: the picture does not move, it is rebuilt — out of the next few syncs, within seconds. During that window the console honestly shows incomplete figures: a stream is already broadcasting but has not yet appeared in the list of active ones.

Nothing is lost in the process — not the archive, not the session accounting, not the settings: only the "right now" picture is incomplete. The practical rule: the first half-minute after a change of holder is not evidence about the state of the cluster, and an alarm about "missing streams" in that window means a warm-up, not a failure.

Layouter¶

The layouter decides which stream runs on which machine. A duplicate of that work is harmful: two layouters over one database would decide at the same time and race each other into the placement journal — streams would start moving between machines for no reason.

So the runs are executed by the holder of the layouter mandate, and its writes are fenced by the epoch of issue. The layouter role, meanwhile, is claimed by every instance of the group: claiming it on one means being left without placement when that particular machine dies.

An instance not holding the mandate is complete in every other respect: it serves the console, the API and streamer sync. Nothing has to be switched on during a failure — the mandate moves by itself.

Schema migrations¶

The one job no mandate guards is applying migrations at start. Two processes starting at once will race each other through them, so the migrate role is claimed by exactly one instance and the rest start after it.

The caveat matters more than it looks: an empty list of roles means not "no roles" but all of them, migrate included. A second instance brought up from a copy of the first one's config turns out to be a second migrator.

Session harvesting¶

Who watched what — the sessions — is not something central receives passively: it pulls them off the streamers itself, over a continuous channel to each streamer.

The work is split by streamer: the channel of one streamer is pulled by one executor at any moment, and different streamers may be pulled by different ones. So the loss of an instance costs not a halt in accounting but the migration of its streamers to a live one.

Downtime loses no records either: a streamer keeps its session journal for several days, and harvesting continues from where it stopped. An hour of downtime means the records arrive later, not that they will not arrive.

Periodic jobs¶

The rest of the background work runs on a schedule: loading the programme guide, cleaning journals by their retention, VOD storage accounting (retiring the storages of vanished machines, expiring unfinished uploads), clearing the status of streamers that have gone silent.

They run in every instance, and that is safe — but for two different reasons.

• Programme guide loading divides its sources with a lock in the database: before a pass an instance takes the lock on a source, and if a neighbour is already updating it the pass is skipped.
• Cleanups and storage accounting need no lock: running them again changes nothing — there is nothing to delete that was already deleted.

There is no need to spread these jobs across instances by hand, nor to switch them off anywhere.

PostgreSQL¶

The database is the only store of state and the only point whose redundancy Catena does not take on itself. What is needed is one logical cluster reachable by every instance at one address; its own fault tolerance is the database's job: a replica with failover, or a managed instance from a provider.

Two independent database servers are not redundancy but two separate installations: their placement and their streamer registry will diverge.

Backups are still needed: a spare protects against the loss of a machine, not against a mistaken deletion — see Backup and restore.

The operator entry point¶

The operator needs one address that finds a live instance by itself: a DNS name with failover, or a load balancer in front of the group.

Streamers do not go through it — they have a list of addresses of their own, and an extra intermediary on the sync path only adds a point of failure.

Streamers¶

Streamers are made redundant not by the number of control-plane instances but by placement: a stream given more than one machine survives the loss of any of them. That is a topic separate from central — see The streamer list.

Scenarios¶

An instance fails¶

Everything happens by itself, with no involvement.

• Streamers move to the next address on their list after one failed request. They need no reconfiguring, and there is no break in broadcasting.
• The mandates of the lost instance move to a live one within seconds. Placement keeps working, new streams are placed as usual.
• The current picture is rebuilt. For half a minute the figures in the console are incomplete — that is a warm-up, not a disappearance.
• The operator works through the same address with the same password. The state lives in the database rather than in the lost machine: the streamer registry, the settings, the placement are all in place.

The one thing a failure really touches is the applying of migrations: if the instance with the migrate role is the one that died, another has to take the role over for the duration, or the next upgrade will have nothing to start from.

An instance comes back¶

Coming back is not an emergency operation and needs no window of downtime. Nothing moves back in the process, and that is a decision rather than an omission.

• The mandates stay with whoever holds them. There is nothing to be gained by taking them away: the holder is working, and a change of holder costs another warm-up of the picture.
• A streamer stays at the address it moved to. While that address answers there is nothing to be gained by changing it.

The group works in any arrangement, so there is no need to restore the original one.

A planned upgrade¶

Instances are upgraded one at a time, and it is the same scenario, only controlled: while one is being upgraded the streamers work with the other. The mandates move twice in the process — once per restart — and both times by themselves. The order and the checks — Upgrade.

Losing the database¶

The control plane stops entirely: there may be any number of instances, but they have one state between them. While the database is unavailable the console and the API serve no requests, and new streams are neither created nor placed. Nobody holds a mandate during that time — the database is what issues them.

The air goes on: streamers are autonomous — they broadcast, record and serve the archive to the viewer without asking central. So losing the database is an outage of the control plane, not of the air.

Recovery is the database's own business (a switch to a replica) or a restore from backup; see Backup and restore.