Statistics¶

Sapsan always collects the full set of counters — your license and your task only determine which channel you read them through. There are four channels:

The selection rule is simple:

• "What is happening right now?" — Admin API or the UI.
• "What happened over the last hours?" — the built-in Prometheus server.
• "What happened last week, and why did it break at night?" — Retroview.
• "I want my own Prometheus, Grafana, and alertmanager" — the extended counters option.

Stream stats in the Admin API¶

Stream list¶

GET /streamer/api-v4/streams/stats returns one element per stream — the channels page in the UI lives on this list, and it is also the right thing to poll from scripts.

Statuses of inputs[] elements:

• active — playing right now;
• ok — a backup, alive as of the last check;
• error — a backup with a recorded problem (text in error);
• unchecked — a backup that has never been checked yet.

This answers the main failover question — "will we actually switch when the primary dies". Backups are checked periodically (recheck_secondary_inputs_interval, see backup sources); alert on error, on unchecked, and on a too-old checked_ago_ms — an unchecked backup cannot be considered working.

Single stream¶

GET /streamer/api-v4/streams/stats/{name} adds diagnostic sections to the same fields — they are absent from the list to keep the list cheap.

The input section — counters of the current input:

The dvr section in the single-stream response is extended with:

Push statistics¶

Outgoing delivery is counted separately from ingest and per push: the key of the statistics is the pair "stream name, push name" — the same one used in the configuration. One stream can go out to five targets, and "the stream sends 40 Mbps" does not answer whether it reaches any particular one of them.

The pushes section¶

GET /streamer/api-v4/streams/stats/{name} (and each item of GET /streamer/api-v4/streams/stats) carries an object keyed by push name:

"pushes": {
  "cdn-main": {
    "status": "running",
    "proto": "udp",
    "url": "udp://127.0.0.1:5500",
    "opened_at": "2026-08-13T11:17:53Z",
    "bitrate_kbit": 2113.4,
    "bytes_total": 19053048,
    "datagrams_total": 14478
  },
  "youtube": {
    "status": "error",
    "proto": "rtmp",
    "url": "rtmp://198.51.100.7/live/secret-key",
    "error": "rtmp connect failed: timeout after 5s",
    "reconnects_total": 22
  },
  "backup-dc": {"status": "disabled", "proto": "srt", "url": "srt://203.0.113.10:9000"}
}

Zero fields are omitted from the answer: an RTMP push has no datagrams, a UDP one has neither frames nor retransmits.

Next to the stream's own counters, as top-level fields of the same answer, are three sums across all pushes — push_bytes_total, push_errors_total, push_reconnects_total. They answer "how much did the stream send at all" without walking the map.

Prometheus series¶

Every push counter is a series of its own, labelled with push:

In the built-in Prometheus server the same counters carry the labels name (the stream) and push; on central, node is added to them. Protocol depth — stream_push_srt_* and stream_push_rist_* — comes in the node scrape together with the extended counters option.

# sending rate per target
rate(stream_push_bytes_total{stream="tv1"}[1m])

# the stream's total outgoing rate across all pushes
sum by (name) (rate(stream_push_bytes_total{name="tv1"}[1m]))

# breaking channels: reconnects over five minutes
increase(stream_push_reconnects_total[5m]) > 0

# losses on the way to the receiver over SRT/RIST
rate(stream_push_retransmitted_packets_total[1m])

How to read the state¶

• status: error with a non-empty error — nothing is being sent, and the reason is named. The push keeps raising itself every five seconds regardless.
• status: running with a zero bitrate_kbit — the connection is there but the data is not: check whether the stream itself is running.
• status: running with a growing reconnects_total — the channel breaks and recovers; look at opened_at, it shows the age of the current connection.
• status: disabled — the push is turned off in the configuration. Its counters are frozen at their last values; nothing is lost.

The console view¶

The console shows the same data. On the Egress tab, a live statistics line runs under the setting of every push: status, rate, errors per second, reconnects, and — for a failed one — the reason.

In the stream list, a failing push lights a mark next to the stream status: the outgoing side is visibly broken without opening the stream.

Built-in Prometheus server¶

Sapsan contains a built-in Prometheus server: the HTTP API /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, /api/v1/label/{name}/values, /api/v1/status/buildinfo on top of its own in-memory store. To Grafana it is a ready-made datasource: add a datasource of the Prometheus type, point it at the server URL — and build graphs without deploying a TSDB. The built-in UI graphs are powered by the same queries.

By-design limitations:

• history is a few hours, in memory; after a restart the store is empty. This is the "what is happening now" channel, not a metrics archive;
• PromQL is supported as a subset: selectors with label matchers, rate/irate/increase, the sum/avg/min/max/count aggregations with by()/without(), arithmetic, offset. An unsupported construct returns an explicit error, not a distorted result;
• the series set is the base one; deep series (per-PID, SRT, RTP) appear with the "extended counters" option.

Querying a single server¶

# stream ingest rate, bytes/s
rate(stream_input_bytes_total{stream="cam1"}[1m])

# input errors over the last 5 minutes
increase(stream_input_errors_total{stream="cam1"}[5m])

# is the archive being written with errors?
increase(stream_dvr_write_segments_failed_total{stream="cam1"}[10m])

# total server ingest traffic
sum(rate(stream_input_bytes_total[1m]))

# memory used by streams, by pipeline part
sum by (part) (stream_memory_bytes)

On a single server the series carry no node label.

Querying central¶

Central and the Sapsan nodes form a single complex: central collects statistics from all nodes, and its built-in Prometheus server serves the very same API. There is one difference, but it runs through everything — the node label:

• every series carries node="node-name" — you can see which node produced it;
• one stream can produce several series: a cluster stream is served in parts on different nodes;
• node appears in /api/v1/labels and in label_values(node) — that is what a Grafana dashboard variable with a node dropdown is made of.

# the stream as a whole, regardless of how many nodes it lives on
sum without (node) (rate(stream_input_bytes_total{stream="cam1"}[1m]))

# ingest traffic of the complex, broken down by node
sum by (node) (rate(stream_input_bytes_total[1m]))

# which nodes the stream lives on right now — check the node label of the result
stream_input_bytes_total{stream="cam1"}

The same dashboard works both against a single Sapsan and against central: aggregate with sum without (node) — on a single server the label does not exist, and the aggregation changes nothing.

Prometheus scrape: your own monitoring¶

For installations with their own monitoring infrastructure Sapsan serves metrics in the Prometheus text format:

• GET /streamer/api-v4/live-metrics — streams and inputs;
• GET /streamer/api-v4/sessions-metrics — playback sessions;
• GET /streamer/api-v4/dvr/metrics — archive disks and catalog;
• GET /streamer/api-v4/runtime/metrics — the process and the allocator.

Differences from the built-in Prometheus server: the history is stored on your side and is limited only by your TSDB retention; the full depth is available, including per-PID/SRT/RTP protocol metrics; alerting is your alertmanager. The counters are monotonic and survive observed restarts — this is also the channel for billing reconciliation on your side.

Extended counters¶

The "extended counters" option unlocks causal and protocol-level detail everywhere at once: in the input and dvr.write sections of the Admin API, in the series of the built-in Prometheus server, and in the Prometheus scrape. The Retroview telemetry always carries these counters regardless of the option — so root-cause analysis is available in Retroview even without it.

Input error causes¶

The aggregate errors answers "the input has a problem"; the causal breakdown answers "which one exactly". Causes come in two families:

• live stream media defects — lost_packets, broken_payload, desync, ts_pat: data arrives, but corrupted;
• source failures — connection_closed, connection_refused, timeout, http_request_error, not_found, denied, decode_error, protocol_error, io_error, other: the source died, and the cause of death is classified.

Classification is derived from the typed error category, not from log text, and covers source open failures too. Every failure also increments the aggregate errors; repeated failures on retries are counted each time — the event rate is the very signal that the source is still dead. Normal stops — reconfiguration, end of stream, preemption by a higher-priority source — are not counted as errors.

Chart in the stream card¶

The "Source errors" panel on the Sources tab of the stream card draws errors per minute as separate lines per cause: one line per cause, and causes with no events do not occupy the legend. Minutes when the stream's input frame rate was zero are highlighted as an outage: you see not only "there were errors" but also "the stream was down at that time".

An unreachable HLS or HTTP MPEG-TS source yields http_request_error on every retry, an SRT handshake timing out — timeout:

Without the "extended counters" option the panel shows a single aggregate errors_total line; the per-cause lines appear together with the option.

The breakdown series¶

In the built-in Prometheus server the breakdown travels as a single series with a cause label — errors_detail_total{name, cause}; on central the series also carry node. Causes with no events produce no series.

# stream errors over 5 minutes, broken down by cause
sum by (cause) (increase(errors_detail_total{name="tv1"}[5m]))

MPEG-TS per PID¶

For every PID of the transport stream: errors_ts_cc (continuity counter violations — the primary transport loss indicator), errors_ts_tei, errors_ts_scrambled (descrambling failed — CAM/key problems), errors_ts_psi_checksum, broken PES, PCR jitter buckets, decoder buffer (HRD) underruns.

Why: this is broadcast-grade monitoring material. An alert on rate(errors_ts_cc) > 0 per PID catches transport degradation before the viewer sees it; PCR jitter and HRD show whether the stream will survive a hardware decoder.

SRT¶

RTT and its variance, losses and retransmits in both directions, too-late drops, buffer fill, keepalive timeouts.

Why: tuning latency to the real link and diagnosing "who loses — us or the remote side". Retransmits grow while RTT is stable — losses on the path; RTT jumps — the link is congested.

RTP/RTSP per channel¶

Lost packets, timestamp jumps and stalls, NACKs, buffer overflows — per RTP channel (video/audio) of a camera.

Why: camera fleet diagnostics — which camera is failing, before the artifact complaints arrive.

DVR write performance¶

segments_written_slow, segments_written_delayed, segments_written_collapsed, segments_discontinuity.

Why: an early "the disk cannot keep up" signal. The slow/delayed share grows under the same traffic — the storage is degrading; discontinuity means gaps you will later see in recorded_hours.

Retroview¶

Retroview is the vendor's cloud monitoring. Once a minute Sapsan sends telemetry — the full counter catalog, including all the extended detail — and Retroview keeps it for months. Nothing needs to be configured on the server: the channel works together with the license.

When to go to Retroview instead of the built-in Prometheus server:

• history and trends — traffic and stream count growth over months, capacity planning;
• postmortems — what happened to the stream at night: telemetry survives restarts and does not depend on server memory;
• usage reconciliation — the counters are sampled over time by an external system, so the bill can be verified even if the servers were restarted;
• root-cause analysis without the extended counters option — telemetry always carries the causal detail;
• curated alerts — product rules instead of homegrown ones.

Recipes¶