Flussonic Media Server documentation

Contents

Flussonic RAID for DVR

Flussonic RAID for DVR (beta version) is an application-level RAID offering high reliability and convenience when writing video data to dozens of disks.

Flussonic RAID has substantial advantages over similar solutions:

  • No need to buy expensive RAID controller hardware for 60 disks, for example. You can use all disks in JBOD mode (Just a Bunch of Disks). You format each disk separately and mount them in the system in a specific directory. After that, you set up Flussonic and it begins to monitor the condition of the disks and allocate the data among them. You can manage data allocation by using configuration options descibed later on this page.

  • Reliability: If any drive fails, data will continue to be recorded to other drives. Only the part of the archive that was written to a failed drive can be lost.

  • Continuous operation: You can add and remove disks from the array while Flussonic is writing to the archive. The changes are applied without restarting Flussonic.

  • Automatic seamless migration of data between RAID disks, which makes it possible to empty a disk while keeping the DVR archive readable and accessible.

  • Automatic data allocation between disks in an array: Flussonic decides on which disk it would be better to write. The amount of data can be bigger than it's possible to write to one disk within acceptable time limits, so Flussonic uses even data allocation among disks. To minimize electricity costs, you can limit the number of disks that can be written on simultaneously.

  • Protection from writing if disks were not mounted. This will prevent all the data from being written to the root partition.

Setting up application-level disk array in Flussonic Media Server

Existing archives cannot be transferred to RAID, you can only start writing new archives to it. To start working with RAID, you'll need to configure the server by adding disk array settings (these are essentially global DVR settings) and specify this array in a stream settings to record the stream into this array.

The setup procedure is different for Flussonic Watcher and for Flussonic Media Server.

The DVR disk array is created at the operating system level when you mount the disks, and then the entire array is managed programmatically by the Flussonic server.

Disk array settings act as global DVR settings. For now, Flussonic allows you to specify array settings in the configuration file /etc/flussonic/flussonic.conf.

First, specify the disk array settings, for example:

dvr my_raid {
  root /dvr/_raid_;
  raid 0;
  metadata idx;

  disk d1;
  disk d2 keep;
  disk d3 migrate;
}

Then specify that a stream must be recorded to the disk array with the archive depth of seven days:

stream channel5 {
  url tshttp://10.0.4.5:9000/streamname;
  dvr @my_raid 7d;
}

This stream will receive the global settings specified for my_raid. You can override some of them in the stream settings.

Configuration example with the full set of options

Disk array has three kinds of settings:

  • Global DVR disk array settings
  • Settings that can be overridden in individual stream settings
  • Options that control the recording process on disks.
dvr my_raid {
  root /dvr/raid_;     # base directory
  raid 0;               # turn on RAID
  active 2;             # write only to 2 disks
  metadata idx;         # path to cached metadata
  no_index;             # do not index streams
  limits 90% 3G 1d;     # limits for archives
  copy /opt/storage;    # copy to /opt/storage
  replicate port=2345;  # replicate (specify on a restreamer)
  schedule 3-6,5-8,23-5;# when to write

  disk d1;              # disk in usual mode (read and write)
  disk d2 keep;         # do not delete the data
  disk d3 readonly;     # read-only mode
  disk d4 abandon;      # do not write, but can delete old data
  disk d5 migrate;      # copy to other disks and remove all data
  disk d6 rescue;       # copy to other disks but keep the data
}

Global DVR disk array settings, which apply only to the DVR on a disk array:

  • root — the base directory where disks are mounted and indexes are located.

  • raid — enables the work with the array (the allowed value is 0). If you enable RAID, it checks for active disks. Flussonic will check the major device of the root path and files in directories and will not allow writing if a directory was not mounted successfully.

  • active — the number of disks to which data will be written. With a large number of disks, to record all at once would be uneconomical (high power costs), so you'd better write only to a few disks simultaneously. If you do not specify this option, all disks with sufficient free space will be written to.

  • metadata — the subdirectory in root for storing cached metadata. You don't need to create this directory manually, it will be created when the configuration is applied. We recommend using SSD for quick access to the archive.

  • disk — the path to a mounted disk. The paths specified in the disk option must be real mount points. For example:

Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--15--disk--1  7.9G  5.7G  1.8G  77% /
/dev/loop0                       196G  4.0G  182G   3% /dvr/_raid_/d1
/dev/loop1                       196G   4.0G  182G   3% /dvr/_raid_/d2

Settings that apply to individual streams (when specified in global DVR settings, they will apply by default to all streams, but you can override them in the settings of a single stream):

  • no_index — turns off indexing for streams.

  • limits — sets limits on the size and depth of the archive. Example: 90% 3G 1d.

  • replicate — sets the replication of a DVR archive. Port is optional.

  • copy — copies the data in parts to another location.

  • schedule — sets the schedule for recording video to archive.

Options for managing the recording process:

  • keep — Flussonic writes the data, but never purges it. Use this option if you need to store the entire archive permanently.

  • readonly — Flussonic does not write data to this disk, it is only possible to read. Use this option in the following cases:

    • a severely weared hard drive with a lot of wear and tear on which it's undesirable to change any data (including writing, deleting files, updating the index, and copying data to another disk). The disk will be used in read-only mode until the data on is expired and purged.
    • a normally working disk that was once recorded in keep mode, if you want to store the data permanently.
  • abandon — Flussonic will no longer write new data on this disk. When the data becomes old, it will delete it. But it updates the index. Use this option when you want to replace a disk, for example, with a larger one.

  • migrate — use this option if you need to extract the disk urgently but cannot lose the data. Flussonic migrtes the data to other drives in the array on the fly and empties the disk. While it does that, the process of archive recording is not interrupted. You'll know that the data has been migrated when there are no more calls to this disk. After you remove the disk, there is no need to restart Flussonic.

  • rescue — use this option in an emergency. It copies all the data from an old disk that is going to fail soon. Unlike the migrate option, this option does not delete data from the disk. The DVR remains accessible. Writing of new data to such a disk will be stopped.

Mounting disks in Linux

Flussonic RAID is a software RAID, it means that disks should be mounted as general ext4 separate disks. You are not limited to use ext4 filesystem but we strongly recommend it if no weighty reasons to use another filesystem.

Here is the real configuration of one of our laboratory servers:

root@dvr:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   9.8T  0 disk /storage/d1
sdb      8:16   0   9.8T  0 disk /storage/d2
sdc      8:32   0   9.8T  0 disk /storage/d3
sdd      8:48   0   9.8T  0 disk /storage/d4
sde      8:64   0 119.2G  0 disk
└─sde1   8:65   0 119.2G  0 part /

root@dvr:~# cat /etc/fstab
# / was on /dev/sda1 during installation
# OS SSD DISK
UUID=5081dd01-6b97-4166-b05c-e9f59476b553 /               ext4    errors=remount-ro 0       1

#sda
UUID=8c5bcc39-8599-4545-a373-f63a441b53b8 /storage/d1   ext4    defaults,nofail,x-systemd.device-timeout=5  0   2
#sdb
UUID=f4888c12-6faa-4ac1-b4fb-e04c3e4ddc31 /storage/d2   ext4    defaults,nofail,x-systemd.device-timeout=5      0       2
#sdc
UUID=7feedd26-feb4-47ad-99b8-ec732cbd87aa /storage/d3   ext4    defaults,nofail,x-systemd.device-timeout=5      0       2
#sdd
UUID=ed41fad5-92fd-4737-87ff-c6a859aeed10 /storage/d4   ext4    defaults,nofail,x-systemd.device-timeout=5      0       2

Key points:

  • OS Disk must be SSD disk.
  • All HDDs for Flussonic array must be mounted to the same root directory.

Flussonic configuration:

dvr raid {
  root /storage;
  raid 0;
  disk d1;
  disk d2;
  disk d3;
  disk d4;
}

Please consult with our local system administrator or our support team if you've any questions about Linux mount or Flussonic RAID configuration.