Skip to content

Monitoring

Sapsan writes structured logs, serves Prometheus-format metrics, and exports traces via OpenTelemetry (OTLP) — it fits the standard observability stack: Prometheus/Grafana, Jaeger/Tempo, Loki.

Logs

The log level is controlled by the RUST_LOG environment variable:

RUST_LOG=info sapsan
RUST_LOG=debug,hls=trace sapsan   # more detail for a specific module

Note

TODO: log format, default destinations in deb/Docker installations.

Prometheus metrics

GET /streamer/api-v4/runtime/metrics serves process metrics in Prometheus format (including the jemalloc allocator). Next to it — live, session, and DVR metrics: /live-metrics, /sessions-metrics, /dvr/metrics (see the Admin API).

Quality counters

What is available for alerting:

  • MPEG-TS ingest: per PID — CC errors, TEI, scrambled packets, PSI CRC errors, broken PES, decoder buffer (HRD) state.
  • SRT: RTT, loss and retransmits both ways, buffer sizes, keepalive timeouts.
  • Sources (LSI): time on primary/backup input, time with no data, retries, backup validity, parameter divergence between inputs.
  • Sessions: opened, denied, authorized, re-auth denials.
  • DVR: disk space, op counters, blob counts, bytes used by the archive, catalog rebuild progress.
  • Pushes and Peeklio agents: connection state, bytes, errors.

Traces

Sapsan exports traces over the OTLP protocol.

Note

TODO: OTLP exporter environment variables (endpoint, sampling), what is covered by traces.

Server health

  • Readiness probe: GET /streamer/api/v3/monitoring/readiness — for Kubernetes and load balancers.
  • Stream state: GET /streamer/api/v3/streams.

Diagnostic tools

The built-in HLS and DASH validators actively check your own (or someone else's) delivery: LL-HLS latency, timeline integrity, ABR ladder alignment.