External stream management¶

Flussonic provides tools for handling static and dynamic streams. It allows you to load static stream configurations and use live locations to publish streams with dynamic names.

Thus, stream names can be:

• known to Flussonic in advance and specified in the configuration file (static streams),
• unknown to Flussonic in advance, but known to the external subsystem before accessing Flussonic (streams with dynamic names).

When the system consists of two or three servers, the mechanisms for static streams and streams with dynamic names work. If the system expands and the number of servers increases, these mechanisms start to break down, and issues arise.

Issues in managing a large cluster of servers¶

If the system includes 20 or more servers and works with many streams, both static and with dynamic names, the following issues arise:

1. It is unclear how to distribute the load between servers effectively.

   With a growing number of different streams, the system needs to run some streams with static configuration and others on client request. The mechanisms for static configuration work when the system is small and consists of one or two servers. When the system expands, the number of servers increases to 20, 30, or more, and the amount of content on these servers becomes even larger, it becomes unclear how to distribute streams between servers effectively. In this case, the usual mechanisms for static streams do not work anymore.

2. It may be necessary to log on to each server in a cluster to make configuration changes.

   A system that manages 20 or more Flussonic servers, capturing TV channels or IP cameras, should store the knowledge that a stream is captured and captured only once.

   The system should also store that a particular stream is captured by a particular server, like as server A. The system should also check from time to time that the stream is active and the signal is captured. If server A stops working, another server should capture the stream, like server B, and store that server B is now capturing the stream. If server B fails, another server has to capture the stream, and system should store the location of the stream. When a new server starts capturing the stream, the system should also check the statuses of the previous servers. If one of them starts working again, it is necessary to either:

   • remove that stream from the stream configuration of all the previous servers and leave it on the current server

   or:

   • return the stream to the server that it was on initially and remove it from the stream configurations of previous servers, avoiding stream capturing twice.

   Thus, if you need to make any changes related to stream configuration, you must bypass all servers in the cluster. If one of these servers is currently down, then when it comes up, it may be running with outdated settings. These settings can corrupt the source.

Flussonic solves the issues of capturing a stream twice, and failing of one of the capture servers with the cluster ingest mechanism (cluster_ingest). This mechanism allows you to automatically capture streams on another server when one of the servers in the cluster fails. It also removes streams from other servers when the initial one recovers. However, this mechanism solves problems in one way without the ability to customize it. So we came up with a solution to manage the configuration of streams using an external configuration backend called config_external.

About config_external¶

config_external is an internal Flussonic mechanism by which the server can download the current configuration of streams from the configuration backend. The configuration backend or the configuration server is an external resource that stores the stream management logic and is connected to the database to store that configuration.

How it works¶

The operation sequence of config_external differs depending on whether separate requests are implemented on the configuration backend to update static streams GET /update_streams_list and handle dynamic streams GET /dynamic_streams_list. By default, all requests are executed via a single method GET /streams.

Learn more about the two cases below.

One method for all streams (default)¶

The config_external works in cycles and updates the configuration every two to three seconds. Each configuration update cycle is as follows:

1. Flussonic requests the list of currently active static streams from the configuration backend via API (method GET /streams) and starts them if they have not started yet.
2. Flussonic calculates the difference between the list of stream names already running on the server and the list of static stream names returned by the configuration backend.
3. Flussonic sends an API request to the configuration backend with the resulting list of stream names (method GET /streams with ?name=.. in the query string) to request configuration for these streams.

1. The configuration backend returns the configuration for the requested list of streams. If the configuration backend has not returned the configuration for some of the requested streams, they are removed from the server automatically.

Separate methods for updating static streams and querying dynamic streams¶

Flussonic requests the list of currently active static streams from the configuration backend via GET /streams method.

If the configuration backend returns X-Config-Server-Separate-Endpoints: true header in response, further requests to the configuration backend are sent as follows:

• The GET /streams method is called periodically, and if there are running streams on Flussonic server that are not included into the GET /streams response, then GET /update_streams_list is called to get configuration or delete such streams (similar to the default behavior).

• If a stream with an unknown (dynamic) name is requested from the Flussonic server, then GET /dynamic_streams_list is called to get the dynamic stream config.

  Several consecutive requests of this kind can be combined into one API call at Flussonic's discretion with stream names listed in the name parameter.

Usage scenarios¶

The configuration backend and config_external replace all mechanisms of managing the Flussonic cluster. They provide the ability to implement such mechanisms on your side.

With config_external, you can develop any logic of stream distribution between the servers in a cluster according to your needs. Powered by the configuration backend, you can implement your version of geo-targeted cluster ingest or the source mechanism for video retransmission. Here are the usage scenarios for you to consider:

Geo-targeted cluster ingest¶

The task is to capture the signal from the source in one country by one of the servers nearby.

The algorithm is as follows:

1. Flussonic requests the stream configuration from the configuration backend.
2. The configuration backend looks through the list of available servers and chooses the ones geographically closest to the source.
3. The configuration backend returns the configuration with a source ingest to one of those Flussonic servers.

Dynamic targeted republishing¶

The goal is to send a publish request to the least loaded transcoder.

You can implement the following operating logic:

1. The client sends a request to the publishing server.
2. The publishing server accesses the configuration backend and requests the stream configuration for the client.
3. The configuration backend looks through the list of available transcoders, identifies the least loaded of them, and returns a stream configuration with a push to the least loaded transcoder to the publishing server. There will be no loss of the first frame when the publishing server sends the stream to the transcoder.

Configuring config_external¶

You can set the URL of the configuration backend in one of the following ways:

• add the config_external global option to the configuration file (/etc/flussonic/flussonic.conf):

config_external https://example.com/config_backend/streams;

• pass the config_external parameter in the API request (see the API Reference):

curl --request PUT --url http://127.0.0.1:80/streamer/api/v3/config \
--data '{"config_external":"https://example.com/config_backend/streams"}' \
--header 'Authorization: Basic base64_encoded_username:password' \
--header 'Content-Type: application/json'

• create an environment variable STREAMER_CONFIG_EXTERNAL and specify the path to the configuration backend as the value. We recommend using the environment variable only for the k8s environment (or other automatic deployment system); in other cases it is more convenient to use the first way of setting the option in the config file.

STREAMER_CONFIG_EXTERNAL=https://example.com/config_backend/streams

With this configuration, Flussonic will make two requests to the configuration backend every two or three seconds:

1. the list of currently active streams that should be running on a server,
2. settings for streams with dynamic names, if any.

See also Config validation to learn how to find out if there are any errors in the configuration.

Managing episodes¶

config_external allows you to implement a mechanism for protecting certain parts of the archive from automatic deletion; you can use it to address the following objectives:

• To provide a nPVR (Network Personal Video Recorder) service, i.e. to save an archive with a recorded TV show broadcasted within a certain time period.
• To save an IP camera archive containing some important data (motion, face recognition, etc.)

We use episodes to protect the recordings. An episode is a set of metadata about the section of the archive including a unique identifier, the start time of the section, optionally the end time, etc.

The info about episodes is stored on a configuration backend server (Flussonic Central by default, see Protecting DVR sections from deletion). When Flussonic gets to a periodic task of the archive cleanup, it queries the configuration backend for a list of episodes for each stream and does not delete the parts of the archive that are covered by the episodes.

To enable the episodes:

1. Send X-Config-Server-Episodes: true header in response to GET /streams.
2. Set episodes_url in Flussonic's DVR configuration.
3. Implement the response to the GET /episodes request on the side of the configuration backend to be returned at the URL you specify in episodes_url.