A central group on separate servers¶

Redundancy explains what an installation with a spare is made of and what the second instance changes. This page is about assembling one: which settings each machine gets, what the streamers are told, and what to check once everything is up.

What is described here is a park of independent servers — one machine per instance. To try the same thing on a single laptop, without renting servers, follow the hands-on lab in Docker: the same group comes up from containers in a few minutes.

What is being deployed¶

An installation with a spare has four parts, and only one of them is Catena itself:

• One PostgreSQL cluster, reachable by every instance at one address. It is the only store of state and the only part whose fault tolerance Catena does not take on itself.
• Two or more central instances — separate machines with the same config, differing in their roles and their advertised address.
• Streamers — separate machines, each given not the address of one instance but the list of the group's addresses.
• An operator entry point — one address with a live instance behind it: a DNS name with failover, or a load balancer.

The rule such an installation starts from: central and broadcasting live on different machines. A local streamer inside the central process will not do here — it shares the fate of the process, and its streams die together with the control plane.

How many of each¶

The config of a central instance¶

Instances of a group differ in exactly two lines — the advertised address and the set of roles. Everything else is identical, the database connection string included.

The first instance:

listeners:
  http:
    - port: 80

central:
  database_url: postgres://central:password@db.example.com:5432/central
  admin_keys:
    - management-key
  roles: [run, migrate, layouter, stats]
  advertise_url: http://central-a.example.com

api_auth:
  login: admin
  password: operator-password

The second one is the same, without the migrate role and with its own address:

central:
  database_url: postgres://central:password@db.example.com:5432/central
  admin_keys:
    - management-key
  roles: [run, layouter, stats]
  advertise_url: http://central-b.example.com

Roles are claims, not assignments¶

A role in the config means "this machine is ready to do this work", not "this machine does it". Which instance actually does the work is decided by PostgreSQL rather than by the config — which is why layouter and stats are claimed by both, and that is neither a mistake nor a double start.

• run — serve requests: the console, the management API, the sync coming in from streamers, the metrics export. Every instance needs it.
• migrate — apply schema migrations at start.
• layouter — execute placement runs.
• stats — hold the cluster-wide picture of what is happening right now.

The one role that must not be left on every instance is migrate. It is not guarded by a mandate: two processes starting at once will race each other through the migrations. So one instance applies the schema and the rest start after it.

This is also the reason not to leave roles empty on the second instance: an empty list does not mean "no roles" — it means all four, migrate included.

The advertised address¶

advertise_url is the address at which the group's other instances reach this machine. It is needed because the picture of what is happening right now is held by one instance while the others reach it over the network: the holder writes down its address, the neighbours read it from there.

A load balancer address will not do — it would lead a neighbour back to itself. The address has to be the address of one specific machine.

The setting may be omitted: an instance without an advertised address works fully and may even hold the picture — but its neighbours will not reach that picture, and they will say so in the log. In the console such a row is marked; see what to check.

Licensing¶

A license is needed by every central process, not one per group: without it the process does not start at all. The key and the declared product are the same on every instance.

Who does what: mandates¶

Work whose duplicate would be harmful is done by one instance in the group — the one that has taken the mandate for it. A mandate lives in PostgreSQL and is issued on a connection of its own, so the death of a process releases it by itself, with no timeouts and no sorting out of stuck executors: the session is gone, the mandate is free.

There are several mandates and they are independent. Central has no single leader: layouter and stats may sit on different machines, and that is a normal state rather than an imbalance.

Every issue carries a number — an epoch. It grows when the holder changes and travels with every write: an instance that wakes up from a pause no longer the holder is refused at the write instead of corrupting the placement after the fact. This is what makes a mandate handover safe and free of manual confirmation.

A change of holder takes seconds — as long as PostgreSQL needs to notice the dead session and a neighbour needs to come for the freed mandate.

An instance holding nothing is neither a spare nor asleep: it fully serves the console, the API and streamer sync. A mandate concerns background work only.

The config of a streamer¶

A streamer addresses the group, not a process, so its config carries a list of addresses:

managed_by:
  name: streamer-01
  url:
    - http://central-a.example.com
    - http://central-b.example.com
  join_token: join-token

The order of the list is a starting preference, not a permanent binding. The streamer sticks to its current address for as long as it answers and moves on to the next one after a failed request. It does not come back on its own: while the new address answers there is nothing to be gained by changing it, and cycling for the sake of evenness would only smear the machine across the group.

The second address is mandatory, not desirable. A broken sync loop is the only sign of a streamer's liveness that central has. A streamer left with a single address that has died counts as offline a minute and a half later, and the layouter then honestly moves its streams to another machine — off a live, broadcasting one. The death of one control-plane instance turns into the migration of half the air.

It is convenient to split the list by halves of the park: give one half the first instance as its first address and the other half the second. In the quiet state the sync load is then spread, and when either instance dies only its own half moves.

The operator entry point¶

The operator needs one address that finds a live instance by itself: a DNS name with failover, or a load balancer in front of the group. A request handler holds no state of its own, so any instance can answer an operator's request.

Streamers must not go through that entry point: they have a list of addresses of their own, and an extra intermediary on the sync path only adds a point of failure.

Deployment order¶

1. Bring up PostgreSQL and create the database.
2. Start the first instance — the one with the migrate role. Wait until it answers requests: the schema is applied before readiness.
3. Start the remaining instances.
4. Open the join window and bring up the streamers, giving each of them the list of the group's addresses.
5. Point the operator entry point at the group.

The package on the instance machines is catena-base, the same one the streamers get: the installation question about an existing PostgreSQL is answered with the address of the shared database, while the roles and the advertised address come from the configuration above. The catena package is not wanted here — it sets the database up on its own machine, and a group needs it apart from the instances.

The order matters only at the first step: the other instances will not come up before the schema, and streamers will not join before a live central.

What to check once it is up¶

Everything worth looking at is in the console, under Cluster → Central.

• The table has as many rows as there are instances. A row appears within seconds of a start. If it does not, the instance never reached the database.
• Every row has its "Advertise URL" filled in. The "not advertised" mark means the neighbours will not reach this instance: it has no advertise_url.
• The "Role claims" column shows the declared roles. Make sure migrate is claimed by exactly one.
• The "Mandates held" column is non-empty on at least one instance. Empty everywhere means no instance claimed the layouter and stats roles, so the group has neither placement nor a current picture.
• The "Build" column is the same in every row. Different versions in one group are a normal state only for the duration of an upgrade.
• The Streamers section shows the whole park from any instance. Half the park visible at one address and the other half at another means the instances do not see each other's picture: check advertise_url.

There is no need to test the handover by hand, and it is not worth doing on a working installation — the hands-on lab exists for that, on a disposable stand.

What this arrangement does not do¶

• It does not make the database redundant. PostgreSQL fault tolerance is the database's own job: a replica with failover, or a managed instance from a provider.
• It does not replace backups. A spare protects against the loss of a machine, not against a mistaken deletion — see Backup and restore.
• It does not touch the air. Streams broadcast and record without central; only the control plane is made redundant, see Streamer autonomy.