Skip to content

Input errors

A single error counter only answers the question "there is a problem on the input". For a headend that is not enough: the cause decides who to call. Lost packets are the network, a refused connection is the partner, a handshake timeout is their encoder or the firewall between you.

So input errors are counted by cause, and every cause is its own line on the chart and its own series in the metrics.

Two families of causes

  • Media defects of a live stream — the data arrives but is damaged: lost_packets, broken_payload, desync, ts_pat.
  • Source failures — there is no data, and the cause of death is named: connection_closed, connection_refused, timeout, http_request_error, not_found, denied, decode_error, protocol_error, io_error, other.

The difference is practical. The first family means the signal is running and has to be fixed at the transport level — the TR 101 290 counters are the next place to look. The second means there is no signal at all, and it is the connection that has to be fixed.

How this is counted:

  • the classification comes from the typed error category, not from the log text — rewording a message does not break the counter;
  • a failure to open a source also lands in the detail, not only a failure of an already running connection;
  • every failure also enters the aggregate errors_total counter, so the sums add up;
  • repeated failures on retries are counted each time: the rate of events is exactly the signal "the source is still dead";
  • orderly stops — reconfiguration, end of stream, pre-emption by a higher-priority input — are not counted as errors.

The chart on the stream card

The Source errors and recoveries panel on the Sources tab draws errors per minute as separate lines by cause. Causes with no events take no place in the legend — the chart shows exactly what is happening.

The minutes in which the input frame rate was zero are highlighted as an outage. That answers a question a single error counter never closes: were these errors on a live signal, or was there no signal at all at that time.

Loss of an SRT source: outage bands and retry timeouts

An unreachable HLS or HTTP MPEG-TS source gives http_request_error on every retry, an SRT handshake failing by timeout gives timeout:

Dead inputs: http_request_error and timeout

Without the extended counters option the panel shows a single aggregate line; the detailed lines appear together with the option.

Recovered losses

SRT and RIST recover some network losses through retransmission. Accepted retransmissions that fill gaps count as recoveries. A separate dotted line, Recovered by retransmission, shows the number of packets recovered per minute.

The recovery line shows losses repaired by the transport, which are absent from the unrecovered-loss counter. Rising recoveries without errors can reveal a deteriorating link before content is lost.

Packets recovered on an SRT link with 2% packet loss

  • Only retransmissions accepted into the buffer that filled a gap are counted. Duplicate, late and rejected retransmissions are not.
  • Recoveries are not part of errors_total and do not affect input failover: they are warnings, and their aggregate is warnings_total.
  • The same line exists for an SRT input that pulls a stream and for an SRT publishing port a stream is sent to.

The metrics series

In the built-in Prometheus server the detail travels as one series with a cause label — errors_detail_total{name, cause}; when there is more than one station, the series also carry node. Causes with no events create no series.

# stream errors over 5 minutes broken down by cause
sum by (cause) (increase(errors_detail_total{name="tv1"}[5m]))

# every input currently seeing connection failures
sum by (name) (rate(errors_detail_total{cause="connection_closed"}[5m])) > 0

Packets recovered by the transport have a dedicated cumulative counter, warnings_recovered_packets_total{name}. It requires the extended counters option; installations with multiple stations also carry the node label. Zero values are also exported:

# how much the network lost and the transport got back, per input
sum by (name) (increase(warnings_recovered_packets_total[5m]))

How to read it

What you see What it means
lost_packets with a live bitrate the network between the source and the station is losing; use the transport counters to find the affected PID
timeout in bursts, outage bands the source does not answer; every burst is another retry
http_request_error as a steady comb the HTTP source is unreachable: address, DNS, firewall or a dead origin
connection_closed on a live stream the partner is tearing the connection down; look at their side
denied, not_found access or path: the key, the streamid, the resource name
desync, broken_payload the data arrives broken — the transport or the encoder on the other side
Recovered by retransmission grows, no errors the link loses packets, but the receive latency is enough; watch the trend
lost_packets appear next to growing recoveries retransmission no longer makes it in time: raise the SRT input's Receive latency (ms) — at least 3–4 RTT to the source

A single error on the chart is normal for a network. The diagnosis comes from the shape: a steady comb of one cause, bursts around the outage bands, growth while the bitrate stays the same.

What next