Skip to content

Built-in Prometheus server

control.observability.prometheus

The central/Sapsan cluster includes a built-in statistics data plane. Sapsan streamers report their current statistics to central; the holder of the stats mandate builds metric history in memory. Queries use this history to serve console graphs without a disk TSDB or a separate metrics server.

A group with several central instances has one statistics plane and one history store. Other instances forward history queries to the mandate holder, so requests through different central instances use the same history. Restarting the holder or transferring the mandate creates a new store: previous samples are not transferred.

The defaults are a 30-second sampling interval and a six-hour retention window. Each series also has a hard limit of 1024 samples, and the store has a limit of 1,200,000 series. These limits bound memory use; they do not guarantee six hours of history in every configuration. For durable history use an external metrics system.

Single-server metrics

A standalone Sapsan streamer exports current counters for an external Prometheus to scrape. This does not provide the built-in historical query API described here: that API belongs to central's cluster statistics plane.

Cluster queries

For Grafana, use a Prometheus datasource with base URL https://central.example/central/api-v4/prometheus and central management authentication. Append the standard Prometheus API paths to this base:

  • Queries: /api/v1/query and /api/v1/query_range.
  • Discovery: /api/v1/series, /api/v1/labels and /api/v1/label/{name}/values.
  • Build information: /api/v1/status/buildinfo.

This URL selects the backend configured by central.stats: built-in history, an external Prometheus or Retroview. The memory and retention limits above apply to built-in history only.

Stream series use name for the stream and server_id for the streamer. The built-in engine supports a subset of PromQL; unsupported constructs return an explicit error.

rate(in_media_bytes_total{name="cam1"}[1m])
sum by (server_id) (rate(in_media_bytes_total[1m]))