Extended counters¶

media.probe.counters

Extended counters¶

The "extended counters" option unlocks causal and protocol-level detail everywhere at once: in the input and dvr.write sections of the Admin API, in the series of the built-in Prometheus server, and in the Prometheus scrape. The Retroview telemetry always carries these counters regardless of the option — so root-cause analysis is available in Retroview even without it.

Input error causes¶

The aggregate errors answers "the input has a problem"; the causal breakdown answers "which one exactly". Causes come in two families:

• live stream media defects — lost_packets, broken_payload, desync, ts_pat: data arrives, but corrupted;
• source failures — connection_closed, connection_refused, timeout, http_request_error, not_found, denied, decode_error, protocol_error, io_error, other: the source died, and the cause of death is classified.

Classification is derived from the typed error category, not from log text, and covers source open failures too. Every failure also increments the aggregate errors; repeated failures on retries are counted each time — the event rate is the very signal that the source is still dead. Normal stops — reconfiguration, end of stream, preemption by a higher-priority source — are not counted as errors.

The breakdown series¶

In the built-in Prometheus server the breakdown travels as a single series with a cause label — errors_detail_total{name, cause}; on central the series also carry node. Causes with no events produce no series.

# stream errors over 5 minutes, broken down by cause
sum by (cause) (increase(errors_detail_total{name="tv1"}[5m]))

MPEG-TS per PID¶

For every PID of the transport stream: errors_ts_cc (continuity counter violations — the primary transport loss indicator), errors_ts_tei, errors_ts_scrambled (descrambling failed — CAM/key problems), errors_ts_psi_checksum, broken PES, PCR jitter buckets, decoder buffer (HRD) underruns.

Why: this is broadcast-grade monitoring material. An alert on rate(errors_ts_cc) > 0 per PID catches transport degradation before the viewer sees it; PCR jitter and HRD show whether the stream will survive a hardware decoder.

SRT¶

RTT and its variance, losses and retransmits in both directions, too-late drops, buffer fill, keepalive timeouts.

Why: tuning latency to the real link and diagnosing "who loses — us or the remote side". Retransmits grow while RTT is stable — losses on the path; RTT jumps — the link is congested.

RTP/RTSP per channel¶

Lost packets, timestamp jumps and stalls, NACKs, buffer overflows — per RTP channel (video/audio) of a camera.

Why: camera fleet diagnostics — which camera is failing, before the artifact complaints arrive.

SDI capture¶

V4L2 capture counters use the stream_input_sdi_ prefix. The base tier includes:

• Line state — sdi_line_state.

• Capture restarts — sdi_producer_restarts_total.

• Frames lost from the card — sdi_boundary_loss_total, based on buffer sequence gaps.

Extended counters add:

• Restart causes — standard changes or missing frames.

• Unsupported ST 337 types and modes — sdi_unsupported_codec_total.

• Unknown VANC data — sdi_vanc_unknown_total.

• CDP data — packet and cc_data pair counts.

• Timecode — accepted, rejected and missing values, and timeline changes.

Names containing producer are retained for compatibility and refer to V4L2 capture. sdi_boundary_loss_total reports losses from buffer sequence numbers; a PTS gap alone does not prove a lost frame. See SDI ingest.

DVR write performance¶

segments_written_slow, segments_written_delayed, segments_written_collapsed, segments_discontinuity.

Why: an early "the disk cannot keep up" signal. The slow/delayed share grows under the same traffic — the storage is degrading; discontinuity means gaps you will later see in recorded_hours.

Quality counters¶

What is available for alerting:

• MPEG-TS HLS for set-top boxes: incomplete ts/mux/ segments and builds that bypass the cache. See MPEG-TS HLS output.
• MPEG-TS ingest: per PID — CC errors, TEI, scrambled packets, PSI CRC errors, broken PES, decoder buffer (HRD) state.
• SRT: RTT, loss and retransmits both ways, buffer sizes, keepalive timeouts.
• Sources (LSI): time on primary/backup input, time with no data, retries, backup validity, parameter divergence between inputs.
• Sessions: opened, denied, authorized, re-auth denials.
• DVR: disk space, op counters, blob counts, bytes used by the archive, catalog rebuild progress.
• Pushes: per push — status with the failure reason, sending rate, bytes, errors, reconnects, SRT/RIST retransmits. See Push statistics.
• Peeklio agents: connection state, bytes, errors.

MPEG-TS HLS output for set-top boxes¶

The mixed ts/mux/ output builds every segment from video and audio, and at the live edge the audio may not make it in time. Two stream counters show what the viewer actually received:

Both series are base tier: in the built-in Prometheus server everyone has them, in the node scrape — with the extended counters option. In the Admin API they come as rates in the mpegts_hls section of the stream stats (incomplete_segments_rate, uncached_segments_rate); the section is present once the stream has had such a segment. Retroview telemetry receives the increment over the window.

# streams whose set-top boxes get segments without audio
rate(stream_mpegts_hls_incomplete_segments_total[5m]) > 0

The counter does not tell "the audio was late" from "there is no audio on the input": check the stream input.