Prometheus metrics¶
control.observability.metrics
Prometheus scrape: your own monitoring¶
The «extended counters» option
The scrape endpoints are only available with the extended counters license option. Without it, use the built-in Prometheus server and Retroview.
For installations with their own monitoring infrastructure Sapsan serves metrics in the Prometheus text format:
GET /streamer/api-v4/live-metrics— streams and inputs;GET /streamer/api-v4/sessions-metrics— playback sessions;GET /streamer/api-v4/dvr/metrics— archive disks and catalog;GET /streamer/api-v4/runtime/metrics— the process and the allocator.
Differences from the built-in Prometheus server: the history is stored on your side and is limited only by your TSDB retention; the full depth is available, including per-PID/SRT/RTP protocol metrics; alerting is your alertmanager. The counters are monotonic and survive observed restarts — this is also the channel for billing reconciliation on your side.
Prometheus metrics¶
GET /streamer/api-v4/runtime/metrics serves process metrics in Prometheus format (including the jemalloc allocator). Next to it — live, session, and DVR metrics: /live-metrics, /sessions-metrics, /dvr/metrics (see the Admin API). The scrape endpoints require the extended counters license option; without it, graphs are served by the built-in Prometheus server (a ready-made Grafana datasource). A full walkthrough of the statistics channels, fields, and alerting recipes is on the Statistics page.
Recipes¶
| Question | Where to look | Problem indicator |
|---|---|---|
| Is the stream alive? | status in the API; rate(stream_input_bytes_total[1m]) in the built-in Prometheus |
status is not running; ingest rate dropped to zero |
| Is the backup ready? | inputs[] in the stats list |
status is error or unchecked; checked_ago_ms far exceeds the check interval |
| Are we living on the backup? | source.url; num_sec_on_secondary_input |
the URL is not the first one from the config; seconds-on-backup keeps growing |
| Is the input degrading? | input.errors, input.retries; causes — extended counters or Retroview |
counters grow while the stream is alive |
| Does the push arrive? | pushes[].status and bitrate_kbit; rate(stream_push_bytes_total[1m]) |
status is not running; the rate is zero while the stream is alive |
| Why is the push not working? | pushes[].error |
the reason in words: connection refused, timeout, publish rejection |
| Is the channel to the receiver breaking? | pushes[].reconnects_total, opened_at |
reconnects grow; opened_at gets younger every few minutes |
| Is someone brute-forcing an SRT delivery with a client map? | node scrape: srt_handshake_attempts_total, srt_handshake_secret_candidates_total |
many secret candidates per attempt — brute force; a couple — a regular viewer |
| Is delivery to a viewer degrading? | srt.health in sessions; the stream_play_srt_session_lost_packets_total series |
verdict degraded or bad; losses grow |
| Is the archive being written? | dvr.to in the list; dvr.write.segments_failed in the single GET |
to lags behind wall-clock time; segments_failed grows |
| Are there gaps in the archive? | dvr.recorded_hours versus the to - from window |
recorded hours are notably fewer than the window width |
| Do the disks keep up? | segments_written_slow/_delayed (option); dvr_disk_free_bytes in the scrape |
the slow-write share grows; free space shrinks faster than expected |
| Is the server healthy? | runtime/metrics: process_rss_bytes; the built-in Prometheus: stream_memory_bytes |
memory grows monotonically without load growth |
| What happened at night? | Retroview | — |
MPEG-TS HLS output¶
Incomplete segments, uncached builds, API fields and alert examples are described under MPEG-TS HLS counters.