Qprober¶

Overview¶

Qprober is an integrated monitoring system within all Mcaster components that measures various event counts across sources and individual data streams within sources, such as MPEG-TS PIDs. The module provides comprehensive stream quality analysis and real-time problem diagnostics.

System Architecture¶

Component Integration¶

Qprober is integrated into all Mcaster components:

• Input modules — monitoring sources and streams
• Processing modules — analysis of processing quality
• Output modules — control of output stream status
• System components — monitoring server resources

Centralized Monitoring¶

Measured counters can be collected centrally across an entire server cluster, but when centralized system is organizationally impossible, Mcaster Appliance package may include built-in visualization of all counters.

Main Metrics¶

Basic Server Indicators¶

• CPU load — processor resource usage
• Disk load — I/O operations and performance
• GPU load — graphics processor usage
• Memory usage — RAM and swap consumption

General Video Stream Indicators¶

• Bitrate — current and average stream bitrates
• Source status — availability and quality of sources
• Frame rate — FPS of input and output streams
• Resolution — current resolution of video streams

Error Monitoring¶

Source Unavailability¶

Qprober tracks:

• Connection loss with sources
• Connection timeouts
• Authentication errors
• Network problems to source

Instrumental MPEG-TS Stream Analysis¶

Analysis according to TR101290 standard includes:

Priority 1 (Critical Errors)¶

• Synchronization — stream synchronization loss
• PAT — Program Association Table errors
• PMT — Program Map Table errors
• CC — Continuity Counter errors

Priority 2 (Important Errors)¶

• PCR — Program Clock Reference errors
• PTS/DTS — timestamp errors
• CAT — Conditional Access Table errors

Priority 3 (Informational Errors)¶

• NIT — Network Information Table errors
• SDT — Service Description Table errors
• EIT — Event Information Table errors

Network Protocol Analysis¶

SRT Analysis¶

• Frame loss — number of lost frames
• Emergency timestamp jumps — sharp time changes
• RTT — Round Trip Time of connection
• Retransmissions — number of retransmitted packets

RTSP Analysis¶

• Packet loss — RTP packet loss statistics
• Jitter — delay variations
• Session errors — RTSP session problems

RTMP Analysis¶

• Frame loss — loss statistics
• Protocol errors — RTMP connection problems
• Buffering — buffer status

Output Stream Monitoring¶

Internal Problems¶

Qprober tracks:

• Encoding errors — transcoder problems
• Multiplexing problems — multiplexer errors
• Buffering errors — buffer overflow

Response to Input Problems¶

The module records:

• Timestamp correction — automatic time corrections
• Counter reset — when drift accumulates
• Source switching — automatic switching to backup

Examples of Tracked Events¶

{
  "timestamp": "2024-01-15T10:30:00Z",
  "source": "input_stream_1",
  "event": "timestamp_correction",
  "details": {
    "drift_accumulated": 1500,
    "correction_applied": 1500,
    "counters_reset": true
  }
}

Problem Diagnostics¶

Determining Problem Source¶

Qprober allows determining the source of incoming problems:

Network Problems¶

• High RTT in SRT connections
• Packet loss in RTMP/RTSP
• Unstable bitrate
• Frequent reconnections

Source Problems¶

• TR101290 errors in MPEG-TS
• Synchronization loss
• Incorrect timestamps
• Encoding problems

Metrics for Analysis¶

The main metrics that need to be studied are obtained through the streams_list API. The streams object contains a list of all streams, where each stream has a stats object with more than a hundred parameters for analysis.

Getting Metrics¶

# Getting list of all streams with metrics
curl -X GET "http://localhost:8080/api/streams_list"

Response Structure¶

{
  "streams": [
    {
      "name": "main_stream",
      "stats": {
        // More than 100 parameters for analysis
        "input": {
          "packets_received": 125000,
          "packets_lost": 5,
          "bitrate": 5000000,
          "fps": 25.0,
          "tr101290": {
            "priority1_errors": 0,
            "priority2_errors": 2,
            "priority3_errors": 5
          },
          "srt": {
            "rtt": 25.5,
            "retransmitted_packets": 15,
            "latency": 35.2
          }
        }
      }
    }
  ]
}

Key Metric Groups¶

Input metrics (stats.input): - packets_received/lost — packet statistics - bitrate/fps — stream quality - tr101290 — MPEG-TS errors by standard - srt/rtmp/rtsp — protocol-specific metrics

Output metrics (stats.output): - packets_sent — sent packets - bitrate/fps — output stream quality - errors — encoding/multiplexing errors

System metrics (stats.system): - cpu_usage — processor load - memory_usage — memory usage - disk_io — input-output operations

Retroview Integration¶

Online Mode¶

It is recommended to use Qprober in online mode together with Retroview service:

• Real-time — instant problem analysis
• Historical data — long-term trend analysis
• Automatic alerts — problem notifications
• Centralized monitoring — single control point

Offline Mode¶

Can also be used in offline on-premises mode:

• Local storage — data remains in infrastructure
• Autonomous operation — independence from external services
• Built-in visualization — as part of Mcaster Appliance

Configuration¶

Basic Settings¶

qprober {
    enabled true;
    sampling_interval 1000;  # milliseconds
    retention_period 86400;  # seconds
    log_level info;
}

Stream Monitoring Configuration¶

stream monitored_stream {
    input udp://239.0.0.1:1234;

    qprober {
        tr101290_analysis true;
        network_metrics true;
        output_metrics true;
        alert_threshold 10;
    }
}

Configuration Parameters¶

API and Interfaces¶

HTTP API¶

# Getting stream metrics
GET /api/qprober/stream/stream_name

# Getting system metrics
GET /api/qprober/system

# Getting TR101290 errors
GET /api/qprober/tr101290/stream_name

# Getting network metrics
GET /api/qprober/network/stream_name

Prometheus Metrics¶

# System metrics
mcaster_cpu_usage{server="server1"} 45.2
mcaster_memory_usage{server="server1"} 67.8
mcaster_disk_io{server="server1"} 125.5

# Stream metrics
mcaster_stream_bitrate{stream="main",server="server1"} 5000000
mcaster_stream_fps{stream="main",server="server1"} 25.0

# TR101290 errors
mcaster_tr101290_priority1{stream="main",server="server1"} 0
mcaster_tr101290_priority2{stream="main",server="server1"} 2
mcaster_tr101290_priority3{stream="main",server="server1"} 5

Data Visualization¶

Built-in Visualization¶

Mcaster Appliance includes built-in visualization:

• Dashboards — real-time key indicators
• Graphs — historical data and trends
• Alerts — critical event notifications
• Reports — detailed analytics

External System Integration¶

Qprober supports integration with:

• Grafana — for advanced visualization
• Prometheus — for metrics collection
• ELK Stack — for log analysis
• Zabbix — for infrastructure monitoring

Usage Recommendations¶

Performance Optimization¶

• Configure collection interval for metrics according to your needs
• Use filtering to reduce load
• Plan storage of historical data
• Monitor resources of Qprober itself

Alert Configuration¶

• Define critical thresholds for your streams
• Configure escalation of notifications
• Use different levels of alert importance
• Test alerts in test environment

Data Analysis¶

• Regularly analyze quality trends
• Correlate problems with external factors
• Document typical problems and solutions
• Plan improvements based on data

Conclusion¶

Qprober represents a powerful monitoring and stream quality analysis system integrated into all Mcaster components. The module provides comprehensive problem diagnostics, from basic system metrics to detailed MPEG-TS stream analysis according to TR101290 standard. The ability for centralized monitoring of server clusters and built-in visualization make Qprober an indispensable tool for ensuring broadcast quality in professional systems.