Manage stream configuration externally: configuration backend and config_external mechanism
Flussonic provides tools for handling static and dynamic streams. It allows you to load static stream configurations and use live locations to publish streams with dynamic names.
Dynamic names are Flussonic stream names unknown beforehand that Flussonic receives from the client. In a large system, stream names are unknown only to the Flussonic server but are known to some external subsystems. The external subsystem generates stream names, stores them, and returns them to the client on request. The client then reaches Flussonic with that name. Read more at Publish by dynamic name.
Thus, stream names can be:
- known to Flussonic in advance and specified in the configuration file (static streams),
- unknown to Flussonic in advance, but known to the external subsystem before accessing Flussonic (streams with dynamic names).
When the system consists of two or three servers, the mechanisms for static streams and streams with dynamic names work. If the system expands and the number of servers increases, these mechanisms start to break down, and issues arise.
Issues in managing a large cluster of servers
If the system includes 20 or more servers and works with a large number of streams, both static and with dynamic names, the following issues arise:
- It is unclear how to distribute the load between servers effectively.
With a growing number of different streams, the system needs to run some streams with static configuration and others on client request. The mechanisms for static configuration work when the system is small and consists of one or two servers. When the system expands, the number of servers increases to 20, 30, or more, and the amount of content on these servers becomes even larger, it becomes unclear how to distribute streams between servers effectively. In this case, the usual mechanisms for static streams do not work anymore.
- It may be necessary to log on to each server in a cluster to make configuration changes.
A system that manages 20 or more Flussonic servers, capturing TV channels or IP cameras, should store the knowledge that a stream is captured and captured only once.
The system should also store that a particular stream is captured by a particular server, like as server A. The system should also check from time to time that the stream is active and the signal is captured. If server A stops working, another server should capture the stream, like server B, and store that server B is now capturing the stream. If server B fails, another server has to capture the stream, and system should store the location of the stream. When a new server starts capturing the stream, the system should also check the statuses of the previous servers. If one of them starts working again, it is necessary to either:
- remove that stream from the stream configuration of all the previous servers and leave it on the current server
- return the stream to the server that it was on initially and remove it from the stream configurations of previous servers, avoiding stream capturing twice.
Thus, if you need to make any changes related to stream configuration, you must bypass all servers in the cluster. If one of these servers is currently down, then when it comes up, it may be running with outdated settings. These settings can corrupt the source.
Flussonic solves the issues of capturing a stream twice, and failing of one of the capture servers with the cluster ingest mechanism (
cluster_ingest). This mechanism allows you to automatically capture streams on another server when one of the servers in the cluster fails. It also removes streams from other servers when the initial one recovers. However, this mechanism solves problems in one way without the ability to customize it. So we came up with a solution to manage the configuration of streams using an external configuration backend called
config_external is an internal Flussonic mechanism by which the server can download the current configuration of streams from the configuration backend. The configuration backend or the configuration server is an external resource that stores the stream management logic and is connected to the database to store that configuration.
How It Works
config_external works in cycles and updates the configuration every two to three seconds. Each configuration update cycle is as follows:
- Flussonic requests the list of currently active static streams from the configuration backend via API (method GET /streams) and starts them if they have not started yet.
- Flussonic calculates the difference between the list of stream names already running on the server and the list of static stream names returned by the configuration backend.
- Flussonic sends an API request to the configuration backend with the resulting list of stream names (method GET /streams with
?name=..in the query string) to request configuration for these streams.
If the list of stream names is large enough, Flussonic will divide that list into parts of about a kilobyte each. Then Flussonic will request configuration for a part of the stream list until the end of list. It reduces the load on the configuration backend and the amount of traffic used.
- The configuration backend returns the configuration for the requested list of streams. If the configuration backend has not returned the configuration for some of the requested streams, they are removed from the server automatically.
We do not recommend using one configuration server to serve all Flussonic servers in a cluster. The server won't be able to handle that number of requests and will be overloaded. To avoid this, we recommend making a full or partial data replication of the database on the local machine. If you use Kubernetes, this can be a sidecar container. This way, even if the connection between the central configuration server and Flussonic is lost, your system will be able to continue running, making your service more reliable.
The configuration backend and
config_external replace all mechanisms of managing the Flussonic cluster. They provide the ability to implement such mechanisms on your side.
config_external, you can develop any logic of stream distribution between the servers in a cluster according to your needs. Powered by the configuration backend, you can implement your version of geo-targeted cluster ingest or the
source mechanism for video retransmission. Here are the usage scenarios for you to consider:
Geo-targeted Cluster Ingest
The task is to capture the signal from the source in one country by one of the servers nearby.
The algorithm is as follows:
- Flussonic requests the stream configuration from the configuration backend.
- The configuration backend looks through the list of available servers and chooses the ones geographically closest to the source.
- The configuration backend returns the configuration with a source ingest to one of those Flussonic servers.
Dynamic Targeted Republishing
The goal is to send a publish request to the least loaded transcoder.
You can implement the following operating logic:
- The client sends a request to the publishing server.
- The publishing server accesses the configuration backend and requests the stream configuration for the client.
- The configuration backend looks through the list of available transcoders, identifies the least loaded of them, and returns a stream configuration with a
pushto the least loaded transcoder to the publishing server. There will be no loss of the first frame when the publishing server sends the stream to the transcoder.
config_external (see the API Reference), create an environment variable
STREAMER_CONFIG_EXTERNAL and specify the path to the configuration backend as the value:
With this configuration, Flussonic will make two requests to the configuration backend every two or three seconds:
- the list of currently active streams that should be running on a server,
- settings for streams with dynamic names, if any.
If the configuration backend does not return the configuration for the requested stream, Flussonic will use the stream configuration from the configuration file stored on disk (
flussonic.conf). Make sure that you do not use both configuration backend and configuration file stored on disk to configure streams. Otherwise, the server will not work correctly.