Monitoring¶

Mcaster always collects the full set of counters. The licence and the task only decide which channel you read them through.

The rule of choice is simple:

• "What is happening now?" — the panel or the API.
• "What happened over the last few hours?" — the built-in Prometheus server.
• "What happened last week, and why did it fall over at night?" — Retroview.
• "I want my own Prometheus, Grafana and alertmanager" — the scrape.

What exactly to look at¶

The counters themselves are explained on the task pages, not here:

• Delivery monitoring — destinations: status, rate, errors, reconnects, retransmits.
• Input errors — failure causes and the minutes with no signal.
• TR 101 290 — transport quality per PID.
• Input redundancy — time on the primary and on the backup.

The built-in Prometheus server¶

A Prometheus server runs inside Mcaster: the HTTP API /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, /api/v1/label/{name}/values over a store of its own in memory. For Grafana it is a ready datasource — add a Prometheus datasource and point it at the station's URL. The same queries feed the panel's charts.

Limits by construction:

• the history is a few hours, in memory; after a restart the store is empty. This is the "what is happening now" channel, not a metrics archive;
• PromQL is supported as a subset: selectors with label matchers, rate/irate/increase, the sum/avg/min/max/count aggregations with by()/without(), arithmetic, offset. An unsupported construct returns an explicit error rather than a distorted result;
• the series set is the basic one; per-PID, SRT and RTP depth arrives with the extended counters option.

# the stream's ingest rate
rate(stream_input_bytes_total{stream="tv1"}[1m])

# input errors over 5 minutes by cause
sum by (cause) (increase(errors_detail_total{name="tv1"}[5m]))

# egress per destination
rate(stream_push_bytes_total{stream="tv1"}[1m])

When there is more than one station, every series carries a node label, and one and the same dashboard works against a single station and against the control plane: aggregate with sum without (node) — on a single machine the label is absent and the aggregation changes nothing.

The scrape for your own monitoring¶

Metrics in the Prometheus text format:

• GET /streamer/api-v4/live-metrics — streams and inputs;
• GET /streamer/api-v4/sessions-metrics — sessions;
• GET /streamer/api-v4/dvr/metrics — archive disks and catalog;
• GET /streamer/api-v4/runtime/metrics — the process and the allocator.

The history is kept by you and limited only by your TSDB's retention; the full depth is available, including per-PID, SRT and RTP metrics; the alerting is your alertmanager. The counters are monotonic and survive the observation of restarts.

Retroview¶

Retroview is the vendor's cloud monitoring. The station sends telemetry once a minute — the full catalogue of counters including all the extended detail — and Retroview keeps it for months. Nothing has to be configured on the station: the channel works together with the licence.

When to go there rather than to the built-in Prometheus server:

• history and trends — load growth over months, capacity planning;
• post-mortems — what the stream was doing at night: the telemetry survives restarts;
• consumption reconciliation — the counters are sampled over time by an external system;
• cause analysis without the extended counters option — the telemetry always carries the detail;
• ready alerts — product rules instead of home-made ones.

Logs¶

The logs are structured, the level is set by the RUST_LOG variable:

journalctl -u mcaster -f

The records carry the stream name and the target module, so filtering by a particular stream works without an external indexer. Traces are exported over OpenTelemetry (OTLP).

Station health¶

• Readiness probe: GET /streamer/api/v3/monitoring/readiness — for load balancers and orchestrators.
• Memory by pipeline part: sum by (part) (stream_memory_bytes) — monotonic growth without a growth in load deserves investigation.