Flussonic RAID for DVR¶

Flussonic RAID for DVR is an application-level RAID offering high reliability and convenience when writing video data to dozens of disks.

Flussonic RAID has substantial advantages over similar solutions:

• No need to buy expensive RAID controller hardware for 60 disks, for example. You can use all disks in JBOD mode (Just a Bunch of Disks). Different disk sizes are allowed. You format each disk separately and mount them in the system in a specific directory. After that, you set up Flussonic and it begins to monitor the condition of the disks and allocate the data among them. You can manage data allocation by using configuration options described later on this page.
• Reliability: If any drive fails, data will continue to be recorded to other drives. Only the part of the archive that was written to a failed drive can be lost.
• Continuous operation: You can add and remove disks from the array while Flussonic is writing to the archive. The changes are applied without restarting Flussonic.
• Automatic seamless migration of data between RAID disks, which makes it possible to empty a disk while keeping the DVR archive readable and accessible.
• Automatic data allocation between disks in an array: Flussonic decides on which disk it would be better to write. The amount of data can be bigger than it's possible to write to one disk within acceptable time limits, so Flussonic uses even data allocation among disks. To minimize electricity costs, you can limit the number of disks that can be written on simultaneously.
• Protection from writing if disks were not mounted. This will prevent all the data from being written to the root partition.

On this page:

• Setting up application-level disk array
• Mounting disks in Linux
• Reading runtime statistics
• Global DVR settings in the web UI
• How to replace a failed disk in Flussonic RAID

Setting up application-level disk array in Flussonic Media Server¶

Existing archives cannot be transferred to RAID, you can only start writing new archives to it. To start working with RAID, you'll need to configure the server by adding disk array settings (these are essentially global DVR settings) and specify this array in a stream settings to record the stream into this array.

The setup procedure is different for Flussonic Watcher and for Flussonic Media Server.

The DVR disk array is created at the operating system level when you mount the disks, and then the entire array is managed programmatically by the Flussonic server.

Disk array settings act as global DVR settings. Flussonic allows you to specify array settings in the configuration file /etc/flussonic/flussonic.conf or in the web UI.

Configure DVR settings in the following order. First, specify the disk array settings, for example:

dvr my_raid {
  root /storage/raid;
  raid 0;
  metadata idx;
  disk volume1;
  disk volume2 keep;
  disk volume3 migrate;
}

Then specify that a stream must be recorded to the disk array with the archive depth of seven days:

stream channel5 {
  input fake://fake;
  dvr @my_raid 7d;
}

This stream receives the global settings specified in dvr my_raid. You will have the opportunity to override some of them in the stream settings.

RAID configuration example with the full set of options¶

Disk array has three kinds of settings:

• Global DVR disk array settings
• Settings that can be overridden in individual stream settings
• Options that control the recording process on disks.

Below is the description of all settings.

Global DVR disk array settings, which apply only to the DVR on a disk array:

• root — the base directory where disks are mounted and indexes are located. Example: root /dvr/raid;
• raid — enables the work with the array (the allowed value is 0). If you enable RAID, it checks for active disks. Flussonic will check the major device of the root path and files in directories and will not allow writing if a directory was not mounted successfully. Example: raid 0;
• active — the number of disks to which data will be written. With a large number of disks, to record all at once would be uneconomical (high power costs), so you'd better write only to a few disks simultaneously. If you do not specify this option, all disks with sufficient free space will be written to. Example: active 2;
• metadata — the subdirectory in root for storing cached metadata. You don't need to create this directory manually, it will be created when the configuration is applied. We recommend using SSD for quick access to the archive. Example: metadata idx;
• disk — the path to a mounted disk. The paths specified in the disk option must be real mount points. For example:

Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--15--disk--1  7.9G  5.7G  1.8G  77% /
/dev/loop0                       196G  4.0G  182G   3% /dvr/_raid_/d1
/dev/loop1                       196G   4.0G  182G   3% /dvr/_raid_/d2

Settings that apply to individual streams (when specified in global DVR settings, they will apply by default to all streams, but you can override them in the settings of a single stream):

• limits — sets limits on the size and depth of the archive. Example: 90% 3G 1d.
• replicate — sets the replication of a DVR archive. Port is optional. Example: replicate port=2345;
• copy — copies the data in parts to another location. Example: copy /opt/storage;
• schedule — sets the schedule for recording video to archive. Example: schedule 3-6,5-8,23-5;

Find the modes for managing the recording process in the API Reference.

Mounting disks in Linux¶

Flussonic RAID is a software RAID, meaning that disks should be mounted as general ext4 separate disks. You are not limited to using the ext4 filesystem but we strongly recommend it if you have no weighty reasons to use another filesystem.

Here is the real configuration of one of our laboratory servers:

root@dvr:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   9.8T  0 disk /storage/d1
sdb      8:16   0   9.8T  0 disk /storage/d2
sdc      8:32   0   9.8T  0 disk /storage/d3
sdd      8:48   0   9.8T  0 disk /storage/d4
sde      8:64   0 119.2G  0 disk
└─sde1   8:65   0 119.2G  0 part /

root@dvr:~# cat /etc/fstab
# / was on /dev/sda1 during installation
# OS SSD DISK
UUID=5081dd01-6b97-4166-b05c-e9f59476b553 /               ext4    errors=remount-ro 0       1

#sda
UUID=8c5bcc39-8599-4545-a373-f63a441b53b8 /storage/d1   ext4    defaults,nofail,x-systemd.device-timeout=5  0   2
#sdb
UUID=f4888c12-6faa-4ac1-b4fb-e04c3e4ddc31 /storage/d2   ext4    defaults,nofail,x-systemd.device-timeout=5      0       2
#sdc
UUID=7feedd26-feb4-47ad-99b8-ec732cbd87aa /storage/d3   ext4    defaults,nofail,x-systemd.device-timeout=5      0       2
#sdd
UUID=ed41fad5-92fd-4737-87ff-c6a859aeed10 /storage/d4   ext4    defaults,nofail,x-systemd.device-timeout=5      0       2

Key points:

• OS Disk must be an SSD disk.
• All HDDs for Flussonic array must be mounted to the same root directory.

Flussonic configuration:

dvr raid {
  root /storage;
  raid 0;
  disk d1;
  disk d2;
  disk d3;
  disk d4;
}

Please consult with our local system administrator or our support team if you have any questions about Linux mount or Flussonic RAID configuration.

Reading runtime statistics¶

The runtime statistics about RAID is now included in the response to the API call dvr_get, Learn more about the call in HTTP v3 API.

The statistics shows the state of disks in the RAID:

• status — whether the disk is mounted or not
• blobs_count — the number of blobs on the disk
• size — the size in bytes
• usage — disk utilization percentage
• io_usage — disk utilization percentage from /proc/devstat.

If migration is taking place, the response shows the speed of migration, the estimated end time, and the time when values were last changed:

• migration_speed — the speed of copying the last blob, in bytes per second
• migration_eta — the estimated migration end time, in UTC seconds
• migration_updated — the time when the values of migration_speed and migration_eta were last updated.

After the migration has been completed, this parameters take the value undefined.

Global DVR settings in the web UI¶

This section describes how to write the archive to Flussonic's apllication-level RAID.

To add global DVR settings including RAID arrays:¶

Go to Config > DVR > Add DVR and define the RAID settings. To add more than one array, use the Add DVR button.

Copy chunks to this location is used for copying the archives of static streams to the specified path.

To apply global settings to a stream and override some of them:¶

In the settings of this stream in the DVR section, click in the Global DVR config field and select the previously created array in the list that appears. In the example, an array my_raid was chosen:

Fields on the DVR page will be filled with values from the selected global configuration (Path, Copy chunks to this location, and so on), and you are free to change them for this stream, if necessary.

The stream's archive will be recorded to the RAID array that you have chosen.

To open global DVR settings, in the stream's DVR section click the Edit DVR configurations link under Global DVR config.

How to replace a failed disk in Flussonic RAID¶

Replacing a failed disk in Flussonic RAID generally consists of the following steps:

• Enable for the disk the "Abandon" mode in the tab "DVR" for the streamer. The description of disk modes can be found in the documentation here:

https://flussonic.com/doc/api/reference/#tag/dvr/operation/dvr_disk_save|body|mode

• At the beginning of the next hour, check that writing to this disk is no longer performed:

"lsof -p pidof streamer | grep 'devicename'"

where devicename is the name of the disk in /dev/.

• Replace the "failed" disk with a new one, and check its name - it should not change when saving the mount point on the server.
• Partition and format the new disk, mount it at the same path as the old disk - check the settings in /etc/fstab and the "dvr watcher" section in "/etc/flussonic/flussonic.conf" or in the "DVR" tab for the streamer in Watcher UI.
• Switch the mode for this new disk in the Flussonic raid from "Abandon" to "Normal" in the tab "DVR" for the streamer.
• At the beginning of the next hour Flussonic will start writing the archive to this new disk.