Built-in Prometheus server¶
control.observability.prometheus
The central/Sapsan cluster includes a built-in statistics data plane. Sapsan streamers report their current statistics to central; the holder of the stats mandate builds metric history in memory. Queries use this history to serve console graphs without a disk TSDB or a separate metrics server.
A group with several central instances has one statistics plane and one history store. Other instances forward history queries to the mandate holder, so requests through different central instances use the same history. Restarting the holder or transferring the mandate creates a new store: previous samples are not transferred.
The defaults are a 30-second sampling interval and a six-hour retention window. Each series also has a hard limit of 1024 samples, and the store has a limit of 1,200,000 series. These limits bound memory use; they do not guarantee six hours of history in every configuration. For durable history use an external metrics system.
Single-server metrics¶
A standalone Sapsan streamer exports current counters for an external Prometheus to scrape. This does not provide the built-in historical query API described here: that API belongs to central's cluster statistics plane.
Cluster queries¶
For Grafana, use a Prometheus datasource with base URL https://central.example/central/api-v4/prometheus and central management authentication. Append the standard Prometheus API paths to this base:
- Queries:
/api/v1/queryand/api/v1/query_range. - Discovery:
/api/v1/series,/api/v1/labelsand/api/v1/label/{name}/values. - Build information:
/api/v1/status/buildinfo.
This URL selects the backend configured by central.stats: built-in history, an external Prometheus or Retroview. The memory and retention limits above apply to built-in history only.
Stream series use name for the stream and server_id for the streamer. The built-in engine supports a subset of PromQL; unsupported constructs return an explicit error.
rate(in_media_bytes_total{name="cam1"}[1m])
sum by (server_id) (rate(in_media_bytes_total[1m]))
Streamer and agent state¶
Central records four series about the cluster state:
host_cpu_usage_ratio{server_id}— streamer CPU load from 0 to 1, while the streamer is connected.node_uptime_seconds{server_id}— streamer uptime in seconds, while it is connected. A restart resets the value;resets()counts restarts.node_status{server_id, status}— the streamer's connection state with central: 1 for the currentstatusvalue, 0 for the others.agent_endpoint_connected{agent_id}— 1 if the agent is connected to central, 0 otherwise.
Values of status:
online— central receives reports from the streamer.connection_lost— no reports for more than 30 seconds; the streamer is still considered operational.offline— no reports for more than 90 seconds. It means central receives no reports, not that the streamer is down.
Use a [65s] window in graphs: without it, a series that is no longer recorded keeps showing its last value for five minutes. Make the resets() window 30 seconds longer than the graph step; the example uses a 5-minute step.
last_over_time(host_cpu_usage_ratio[65s]) * 100
sum by (status) (last_over_time(node_status[65s]))
resets(node_uptime_seconds[330s])
count(last_over_time(agent_endpoint_connected[65s])) - sum(last_over_time(agent_endpoint_connected[65s]))