Trying a central group in Docker¶

Redundancy explains how a central group is built, and A central group on separate servers explains how to deploy one. This page is about seeing it work with your own eyes: on one machine, out of containers, in ten minutes and without renting a single server.

The stand is disposable. On it you can — and should — do what one does not do to a working installation: kill processes mid-flight and watch what comes of it.

What you need¶

• Docker with compose — nothing else to install.
• A license key. Every process needs one: both centrals and both streamers.
• An image with the central group. The lab needs a build whose console has a Cluster → Central section. If that section is missing, the image predates the central group and the stand will not come together.
• Free ports 8081 and 8082 on the local machine.

What the stand is made of¶

Five containers:

• postgres — one database for the whole group.
• central-a and central-b — two control-plane instances over that database, with consoles on ports 8081 and 8082.
• streamer-1 and streamer-2 — two streamers split between the instances: the first goes to central-a, the second to central-b.

Splitting the streamers between instances is not decoration, it is the point of the stand. The picture of what is happening right now is one per group, and the whole park has to be visible at either of the two addresses even though the machines report to different ones.

Download docker-compose.yaml into an empty directory. Below are the parts of it that it was written for; the rest is ordinary plumbing.

The first instance claims all four roles, migrate included:

central:
  admin_keys:
    - lab-admin-key
  roles: [run, migrate, layouter, stats]
  advertise_url: http://central-a

The second one is the same without migrate: the schema is applied by one process, or two of them race each other through the migrations.

central:
  admin_keys:
    - lab-admin-key
  roles: [run, layouter, stats]
  advertise_url: http://central-b

The layouter and stats roles are claimed by both, and that is not a double start: a role is a claim, and the executor is chosen by a mandate.

A streamer is given not the address of an instance but the list of the group's addresses; the order is the order of preference:

managed_by:
  name: streamer-1
  url:
    - http://central-a
    - http://central-b
  join_token: ${JOIN_TOKEN}

The second streamer has the same list in the opposite order.

Bringing it up¶

The stand does not come up with a single docker compose up: the join window is opened by a live central, and the join token exists only after it starts.

First the database and both control-plane instances:

export CATENA_VERSION=latest
export LICENSE_KEY='your key'
docker compose up -d postgres central-a central-b

Wait for the console to open at http://127.0.0.1:8081 and log in as admin with the password pass. On the first login the console asks for a named account — create one.

Open Cluster → Streamers, press Permit join and copy the join token. Bring the streamers up with it:

JOIN_TOKEN=jt-... docker compose up -d streamer-1 streamer-2

Both machines appear in the streamer list within seconds. Close the join window afterwards — the park is assembled.

Create a few streams, so that the cluster has something to place. The syntetic input is the built-in generator; the stand needs no external source:

for i in 1 2 3 4 5 6; do
  curl -s -X PUT -H 'Authorization: Bearer lab-admin-key' \
    -H 'Content-Type: application/json' \
    -d '{"inputs":[{"syntetic":{}}]}' \
    "http://127.0.0.1:8081/central/api-v4/streams/configs/cam$i"
done

The group as a whole¶

Open Cluster → Central.

The table has two rows — the entire group — and every instance shows it identically: the registry lives in the shared database.

• Role claims are almost the same on both: migrate is claimed only by central-a, everything else by both machines.
• Mandates held is filled in on one row only. That is the answer to "who is in charge here": nobody is, there are holders of individual mandates.
• The second row says none — serves requests only. Such an instance is neither a spare nor asleep: it fully answers the console, the API and the streamers; it just is not the one doing background work right now.

On your stand the mandates may sit differently — both on one machine, or one on each. Mandates are independent and their arrangement depends on who got there first; there is no wrong arrangement here.

The park is visible at either address¶

Without leaving the instance, open Cluster → Streamers.

Both machines are on air, although only one of them reports to this instance. The other one sends its sync to the neighbour and is visible here because the picture of right now is one per group: an instance without the stats mandate reaches it over the network — at exactly that advertised address.

Open the same page at http://127.0.0.1:8082 and you will see the same thing. This is the check the split of streamers between instances exists for.

Killing the mandate holder¶

Now for the thing the stand is disposable for. Kill the instance holding the mandates — in the screenshots above that is central-b:

docker compose stop central-b

Go back to Cluster → Central on the live instance and wait a few seconds.

Three things have changed:

• The row of the killed instance did not disappear, it is marked: "silent — the process has most likely died". The group remembers its membership, not just its living members.
• The mandates moved to the live instance. Nobody handed them over: a mandate lives on a connection of its own to PostgreSQL and is released together with the death of that session, and the neighbour comes for the freed one by itself.
• The epoch number grew. It grows on every change of holder and travels with every write: an instance that wakes up no longer the holder is refused at the write instead of corrupting the placement after the fact.

The handover takes seconds. There is nothing to confirm by hand, and no way to.

The air never noticed¶

Open Streams.

All six streams are running. Half of them live on the streamer that was reporting to the killed instance: it moved to the second address on its list after one failed request, and central never even had time to consider it lost.

This is exactly why the second address on the list is mandatory. Remove it, repeat the experiment, and a minute and a half later you will watch the layouter move streams off a live machine, because it stopped showing signs of life.

Bringing the instance back¶

docker compose start central-b

The row lights up again, but nothing moves back — and that is a decision rather than an omission.

• The mandates stay with the live instance. There is nothing to be gained by taking them away: the holder is working, and a change of holder costs a new incarnation of the current picture.
• The streamer stays at the address it moved to. While that address answers there is nothing to be gained by changing it; the order of the list is a starting preference, not a permanent binding.

The original arrangement can be restored with a restart — that is what a disposable stand is for — but production does not need it: the group works in any arrangement.

Taking the stand down¶

docker compose down -v

The -v flag removes the volumes as well: the database, the streamers' state and their identity in the cluster. Without it the next start inherits half-foreign state.

What this stand does not show¶

• A database failure. Here the database is single and has no replica: kill it and the control plane stops entirely while the air goes on. That is true, but database redundancy is the database's own job.
• The network. Every container is on one docker network, with no loss and no latency between them.
• Load. Six synthetic streams say nothing about how the group behaves on a real park.
• The operator entry point. On the stand there are two of them, one per instance; in production one address with failover is what is needed.