Dynamic Chat Rooms

Flussonic Media Server has a mechanism for creating a dynamic mosaic that shows multiple streams as one with video and audio mixing. An example of using such a mosaic is creating your own video or audio chat from streams coming to Flussonic server from client devices.

The mosaic is called dynamic because participants (published video streams) can be added and removed programmatically while the chat is running.

The mosaic is designed for streams-publications from client cameras with the following characteristics: video — H.264, audio — Opus. The mosaic is based on the existing in Flussonic publishing and playing via WebRTC. It is played in our WebRTC player.

The mosaic stream receives audio (and video) frames from the participating streams. Then it mixes audio is mixed — the track all is for the audience, all the participants it can be heard; the track STREAM_NAME (name of a participant) — for this participant, you can hear all the participants, except STREAM_NAME.

Video frames are displayed as a grid in a single shared graphic buffer of a certain size, and this buffer is H.264 encoded and delivered as a single frame.

In future versions, it is planned to add settings to adjust the size of the graphic buffer and the parameters of the grid that shows a mosaic.

Configuration and usage

One mosaic — one chat room. To enable a dynamic chat room follow these steps:

1) In the Flussonic configuration file, add the stream that will contain the mosaic of published streams:

stream ROOM_NAME  {
  input mosaic2:// bitrate=128;
}

The following options are available for the mosaic:

• (optional) disable_video=true/false (true by default) — show the video from cameras of chat participants.

• (optional) bitrate, samplerate, samples — parameters of the Opus audio codec: (the default values are 64k, 48000, 960, consequently).

Published streams from client devices that publish to the chat can be added dynamically using the API, manually adding them to the configuration is not necessary.

2) Then use the API to set the mosaic settings:

http://user:pass@flussonic.url/flussonic/api/dynamic_mosaic/

• Add a chat:

curl -X POST -d '{"command" : "add_room", "args" : ["ROOM_NAME"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/'

• Add a chat, working with audio with the follow parameters: bitrate=64 samplerate=4800 samples=480:

curl -X POST -d '{"command" : "add_room", "args" : ["ROOM_NAME","64","48000","480"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/'

• Statistics:

curl -X POST -d '{"command" : "stats", "args" : []}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/'

• Delete a chat:

curl -X POST -d '{"command" : "remove_room", "args" : ["ROOM_NAME"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/'

• List rooms:

curl -X POST -d '{"command" : "list_room", "args" : []}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/'

An example of using the API is shown in the demo application that comes with Flussonic (see below).

3) Using a separate room API, a client device can join this chat. In this case, a regular publishing stream (input publish://) from the client's camera will be created in the Flussonic configuration, and this stream will be added to the mosaic.

• Add a participant

curl -X POST -d '{"command" : "add_stream", "args" : ["STREAM_NAME"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/room'

Returns JSON code similar to {"track_id" : "a3"} — this is the number of the audio track which must be specified on the client player's side in order to hear the chat room.

• Participant leaves

curl -X POST -d '{"command" : "remove_stream", "args" : ["STREAM_NAME"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/room'

Returns the stream's audio track number.

• Statistics

curl -X POST -d '{"command" : "stat", "args" : []}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/room'

Returns JSON code similar to {"count" : 4, "clients" : ["all", "STREAM_NAME", ...]}

• The mute video function is not used yet because Flussonic 20.08 supports the audio-only mosaic.

curl -X POST -d '{"command" : "mute_video", "args" : ["STREAM_NAME"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/room'

Returns the stream's audio track number.

• The mute audio function is implemented both on the server side and on the client (player) side. In the demo app, the client-side mute-audio is demonstrated.

curl -X POST -d '{"command" : "mute-audio", "args" : ["STREAM_NAME"]}' 'http://user:pass@FLUSSONIC-IP/flussonic/api/dynamic_mosaic/room'

Returns the stream's audio track number.