Statistics¶

Sapsan always collects the full set of counters — your license and your task only determine which channel you read them through. There are four channels:

The selection rule is simple:

• "What is happening right now?" — Admin API or the UI.
• "What happened over the last hours?" — the built-in Prometheus server.
• "What happened last week, and why did it break at night?" — Retroview.
• "I want my own Prometheus, Grafana, and alertmanager" — the extended counters option.

Stream stats in the Admin API¶

Stream list¶

GET /streamer/api-v4/streams/stats returns one element per stream — the channels page in the UI lives on this list, and it is also the right thing to poll from scripts.

Statuses of inputs[] elements:

• active — playing right now;
• ok — a backup, alive as of the last check;
• error — a backup with a recorded problem (text in error);
• unchecked — a backup that has never been checked yet.

This answers the main failover question — "will we actually switch when the primary dies". Backups are checked periodically (recheck_secondary_inputs_interval, see backup sources); alert on error, on unchecked, and on a too-old checked_ago_ms — an unchecked backup cannot be considered working.

Single stream¶

GET /streamer/api-v4/streams/stats/{name} adds diagnostic sections to the same fields — they are absent from the list to keep the list cheap.

The input section — counters of the current input:

The dvr section in the single-stream response is extended with:

Built-in Prometheus server¶

Sapsan contains a built-in Prometheus server: the HTTP API /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, /api/v1/label/{name}/values, /api/v1/status/buildinfo on top of its own in-memory store. To Grafana it is a ready-made datasource: add a datasource of the Prometheus type, point it at the server URL — and build graphs without deploying a TSDB. The built-in UI graphs are powered by the same queries.

By-design limitations:

• history is a few hours, in memory; after a restart the store is empty. This is the "what is happening now" channel, not a metrics archive;
• PromQL is supported as a subset: selectors with label matchers, rate/irate/increase, the sum/avg/min/max/count aggregations with by()/without(), arithmetic, offset. An unsupported construct returns an explicit error, not a distorted result;
• the series set is the base one; deep series (per-PID, SRT, RTP) appear with the "extended counters" option.

Querying a single server¶

# stream ingest rate, bytes/s
rate(stream_input_bytes_total{stream="cam1"}[1m])

# input errors over the last 5 minutes
increase(stream_input_errors_total{stream="cam1"}[5m])

# is the archive being written with errors?
increase(stream_dvr_write_segments_failed_total{stream="cam1"}[10m])

# total server ingest traffic
sum(rate(stream_input_bytes_total[1m]))

# memory used by streams, by pipeline part
sum by (part) (stream_memory_bytes)

On a single server the series carry no node label.

Querying central¶

Central and the Sapsan nodes form a single complex: central collects statistics from all nodes, and its built-in Prometheus server serves the very same API. There is one difference, but it runs through everything — the node label:

• every series carries node="node-name" — you can see which node produced it;
• one stream can produce several series: a cluster stream is served in parts on different nodes;
• node appears in /api/v1/labels and in label_values(node) — that is what a Grafana dashboard variable with a node dropdown is made of.

# the stream as a whole, regardless of how many nodes it lives on
sum without (node) (rate(stream_input_bytes_total{stream="cam1"}[1m]))

# ingest traffic of the complex, broken down by node
sum by (node) (rate(stream_input_bytes_total[1m]))

# which nodes the stream lives on right now — check the node label of the result
stream_input_bytes_total{stream="cam1"}

The same dashboard works both against a single Sapsan and against central: aggregate with sum without (node) — on a single server the label does not exist, and the aggregation changes nothing.

Prometheus scrape: your own monitoring¶

For installations with their own monitoring infrastructure Sapsan serves metrics in the Prometheus text format:

• GET /streamer/api-v4/live-metrics — streams and inputs;
• GET /streamer/api-v4/sessions-metrics — playback sessions;
• GET /streamer/api-v4/dvr/metrics — archive disks and catalog;
• GET /streamer/api-v4/runtime/metrics — the process and the allocator.

Differences from the built-in Prometheus server: the history is stored on your side and is limited only by your TSDB retention; the full depth is available, including per-PID/SRT/RTP protocol metrics; alerting is your alertmanager. The counters are monotonic and survive observed restarts — this is also the channel for billing reconciliation on your side.

Extended counters¶

The "extended counters" option unlocks causal and protocol-level detail everywhere at once: in the input and dvr.write sections of the Admin API, in the series of the built-in Prometheus server, and in the Prometheus scrape. The Retroview telemetry always carries these counters regardless of the option — so root-cause analysis is available in Retroview even without it.

Input error causes¶

errors_lost_packets, errors_broken_payload, errors_desync, errors_ts_pat, errors_decoder_reset, errors_dropped_frames, resync and reorder counters.

Why: to move from "the input has errors" to "who is to blame". errors_lost_packets growing — the network loses packets; errors_broken_payload and errors_desync — the source corrupts the stream; errors_decoder_reset — codec-level stream problems. Without this detail you only see the aggregate errors.

MPEG-TS per PID¶

For every PID of the transport stream: errors_ts_cc (continuity counter violations — the primary transport loss indicator), errors_ts_tei, errors_ts_scrambled (descrambling failed — CAM/key problems), errors_ts_psi_checksum, broken PES, PCR jitter buckets, decoder buffer (HRD) underruns.

Why: this is broadcast-grade monitoring material. An alert on rate(errors_ts_cc) > 0 per PID catches transport degradation before the viewer sees it; PCR jitter and HRD show whether the stream will survive a hardware decoder.

SRT¶

RTT and its variance, losses and retransmits in both directions, too-late drops, buffer fill, keepalive timeouts.

Why: tuning latency to the real link and diagnosing "who loses — us or the remote side". Retransmits grow while RTT is stable — losses on the path; RTT jumps — the link is congested.

RTP/RTSP per channel¶

Lost packets, timestamp jumps and stalls, NACKs, buffer overflows — per RTP channel (video/audio) of a camera.

Why: camera fleet diagnostics — which camera is failing, before the artifact complaints arrive.

DVR write performance¶

segments_written_slow, segments_written_delayed, segments_written_collapsed, segments_discontinuity.

Why: an early "the disk cannot keep up" signal. The slow/delayed share grows under the same traffic — the storage is degrading; discontinuity means gaps you will later see in recorded_hours.

Retroview¶

Retroview is the vendor's cloud monitoring. Once a minute Sapsan sends telemetry — the full counter catalog, including all the extended detail — and Retroview keeps it for months. Nothing needs to be configured on the server: the channel works together with the license.

When to go to Retroview instead of the built-in Prometheus server:

• history and trends — traffic and stream count growth over months, capacity planning;
• postmortems — what happened to the stream at night: telemetry survives restarts and does not depend on server memory;
• usage reconciliation — the counters are sampled over time by an external system, so the bill can be verified even if the servers were restarted;
• root-cause analysis without the extended counters option — telemetry always carries the causal detail;
• curated alerts — product rules instead of homegrown ones.

Recipes¶