Flussonic API design principles¶

Table of contents:

• General information about the API design
• Principles of API design
• Authentication and authorization
• OpenAPI support

• Collections

  • HTTP methods for accessing collections
  • Response structure
  • Filtering collections
  • Sorting collections
  • Limiting collection rows (cursors)
  • Limiting the field set of the result

• Creating and updating (upsert)

• Reading objects
• Removing objects

General information about the API design¶

We at Flussonic provide various products and services, such as Media Server, Watcher, Iris, Retroview and so on.

This page provides up-to-date information on the HTTP API management principles concerning our systems.

Principles of API design¶

The Flussonic HTTP API is designed for other applications to access our systems and interact with them. Here are the core principles that we try to follow:

• Focus on industry standards and generally accepted practices while maintaining the principle of reasonableness.
• Organizing the API around REST + JSON.
• Using OpenAPI 3.0 format (formerly Swagger) for a machine-readable API description.
• API-first approach. This means that we design the API before generating the code based on it using the openapi_handler that we developed to support the approach and made available to the public.
• Strict compliance with the API schema returned in the payload data. No fields that are not in the schema can be received from the service.
• Flexible compliance with the API schema to the data sent in payload. Attempting to send a known field in an incorrect format will result in an error. Unknown fields in the payload will be ignored.
• Convenience and simplicity in creating both simple and complex requests (for example, a simple request for streams list and a complex request with multiple filters).
• Consistency in terminology among the systems (the name of an entity in one system must be the same among all the others).
• Standardized access methods to different systems. There might be a collection of one authorization backend on the server and millions of session records in the browsing history storage service. Standardized API should provide access to such systems.
• Preference for HTTP. Access to data on WebSockets is poorly standardized and hard to monitor.
• Obtaining a reasonably limited amount of data for each sample. So the client will not receive a big amount of records by default.
• Supporting the work with a dynamic data type.
• Support of GET requests and declared parameters in the query string to create simple access requests to large collections.
• Idempotent API, i. e. running the same request multiple times produces the same result.
• Supporting work with complex objects, nested subobjects, and collections.

Authentication and authorization¶

With Flussonic API, you can retrieve data and manage certain Flussonic features over HTTP.

To enable the protection for data retrieval requests use the view_auth user password; directive, and for modification of a state and settings requests with the edit_auth user password; directive in the configuration file /etc/flussonic/flussonic.conf.

For more information about the authorization in Flussonic see: Authorization.

To get an access to Flussonic HTTP API functionality with enabled authentication use a login and password according to HTTP Basic Auth.

Authorization for the edit_auth user password; looks as follows:

GET /streamer/api/v3/streams HTTP/1.1
Host: FLUSSONIC-IP
Authorization: Basic dXNlcjpwYXNzd29yZA==

HTTP 200 OK
Date: Sun, 19 Sep 2021 19:40:22 GMT
Content-Type: application/json
...

If you need to use Bearer auth, use:

GET /streamer/api/v3/streams HTTP/1.1
Host: FLUSSONIC-IP
Authorization: Bearer dXNlcjpwYXNzd29yZA==

HTTP 200 OK
Date: Sun, 19 Sep 2021 19:40:22 GMT
Content-Type: application/json
...

OpenAPI support¶

Systems that support the principles mentioned in this document provide service descriptions according to the machine-readable OpenAPI 3.1 specification, which evolved from Swagger and JSON Schema.

OpenAPI defines a standard, language-agnostic interface, which allows formalizing a list of service methods and the input and output data formats.

To retrieve the Flussonic API schema, use the following command provided that your Flussonic server is up and running:

curl http://FLUSSONIC-IP:8080/streamer/api/v3/schema

We also provide public API references as listed on the API overview page. You can get the schema from any of those API Reference URLs by adding .json in the end, for example:

https://flussonic.com/doc/api/reference.json

The example of code below shows the way you can access a list of streams with React Query and openapi-client-axios:

import React from 'react';
import { useQuery } from 'react-query'
import OpenAPIClientAxios from 'openapi-client-axios';


const api = new OpenAPIClientAxios({
  definition: 'http://localhost:8080/streamer/api/v3/schema',
  axiosConfigDefaults: {
    headers: {
      'Authorization': 'Basic dXNlcjpwYXNzd29yZA==',
    },
  },  
});


function Streams() {
  const { isLoading, error, data } = useQuery({
    queryKey: "streams", 
    queryFn : () => {
      return api.init()
        .then(client => client.streams_list({}))
        .then(res => res.data);
    },
    keepPreviousData: true,
    refetchInterval: refetchInterval,
  });

  if(isLoading) return <div>Loading</div>;
  return <div>
    {data.streams.map((stream) => <div key={stream.name}>{stream.name}</div>}
  </div>;
}

The program above reads the API schema, retrieves a list of endpoints and creates a set of functions from it.

In the example above, streams_list is taken from the schema and created programmatically.

Public and private API¶

The response to a request may return fields that are not described in the schema. Such fields are part of the private API. Ignore them like, for example, deprecated fields.

We use the private API to introduce experimental fields that can change without notice or backwards compatibility. When we are confident that such experimental field works as expected, you will see its description in the public API.

The request body may also include private parameters. Other fields that do not relate to either the private or public API will be ignored by Flussonic.

Deprecated fields¶

Some fields in Flussonic API are marked as deprecated: true. This means that we decided to get rid of this field after some time and use some new field instead.

Deprecated fields usually have x-delete-at parameter where we specify Flussonic version in which we are planning to delete this field. For example, x-delete-at: 23.02 means that the field is planned for deletion in Flussonic version 23.02. If you are using this field, and don’t want it to be deleted, you can contact us before the specified version and discuss leaving the field.

Collections¶

Almost all objects are organized in collections (similar to SQL tables). The network's access to the system to read the data is mostly determined by the way objects are read from the collection.

For the fast-running user interfaces and predictably running programs, it is necessary to:

• be able to get the minimum required data set,
• be able to get all the data from the collection with certain reliability.

These two requirements are conflicting. The requirement to limit the data set implies getting only a certain number of objects from the collection, for example, 50 streams out of 2000 on the server.

When the client needs to get the entire list of streams, he will have to make several requests to the server. There is a chance that when new streams appear during the two consecutive requests, these new streams will be missed in the selection.

For predictable access to a subset of the collection, we describe and implement a language that defines the following actions on the collection:

• filtering (similar to SQL WHERE),
• sorting (similar to SQL ORDER BY),
• limiting the number of elements (similar to SQL LIMIT),
• additional cursor filtering (similar to SQL OFFSET),
• limiting the set of fields (similar to SQL SELECT)

HTTP methods for accessing collections¶

The standard way to request a reading from a client to a server is using the query string language and the 'GET' method. It is considered that access that does not involve any modifications is carried out using a GET request (including caching).

When using a query string language, there are two approaches: to use a standard key=value approach or use non-standard delimiters. The second case may cause a number of challenges:

1. Too complex queries with nested conditions. Writing such conditions in a query string will look extremely clumsy and heavy, which contradicts our core principle — the convenience and simplicity of creating requests.
2. Incorrect query processing when using additional separators in subsets. In this case, it is necessary to remember the set of possible characters. For example, comma (",") or hyphen ("-") are not used in field names and are not processed in the query string. The ampersand ("&"), in contrast, is a special character for the query string and, if it is not escaped, issues may arise when the request is to be processed.

For example, some companies use the characters - and + in the sorting request to indicate the sorting direction. The + character is a special character and is treated as space when decoding a query string. Therefore, you either need to escape it using %2B, or use a non-standard parser to avoid possible issues with query execution. The use of non-standard parsers makes it difficult for standard libraries to create a request.

The problems described above intend to illustrate the complexity of using a variety of conditions and versions of the SQL language into a limited HTTP query string language (query string language).

A more complex option is to transmit the request language in JSON format in the body of the POST request. This approach looks like an inevitable solution when using complex nested conditions.

Response structure¶

When accessing the collection Flussonic returns a JSON-encoded response with the following fields:

{
  "ITEMS": [...],
  "next": ...,
  "prev": ...,
  "estimated_count": ...,
  "timing": ...
}

ITEMS field is replaced with the name of the requested collection, i. e. streams for streams and sessions for sessions.

• next and prev are the cursor values (the next and the previous, respectively).
• estimated_count is an approximate number of elements in a collection.
• timing is a service object with various access times. It is not described as it may be changed or deleted in the future.

Filtering collections¶

To retrieve filtered responses, add the parameters in a URL query string.

Filtering by a value¶

To filter by a value (for example, provider), run the following command:

curl http://FLUSSONIC-IP:8080/streamer/api/v3/streams?provider=Sky

To check if elements are in the list:

provider=Sky,Canal,CNN (values are separated by a comma)

To impose a condition on the nested object field:

stats.alive=true

Filtering by a condition¶

It is also possible to use conditions for comparing field values during filtering. We decided to use special suffixes (like _lt, _gt, and so on) for coding such conditions. For example, the condition WHERE stats.delay < 5000 turns into stats.delay_lt=5000.

We have defined a fixed set of suffixes, providing two non-overlapping sets with the existing set of field names among all our systems. It allows us to provide features to create access requests.

Here is a list of supported suffixes:

Several filters specified in the query string are applied sequentially to the collection, similar to AND queries in SQL:

curl  http://FLUSSONIC-IP:8080/streamer/api/v3/streams?stats.bitrate_gt=4000&stats.clients_count_lt=10

Sorting collections¶

Collection sorting is needed for displaying in the web interface Flussonic UI and retrieving a predictable page-oriented selection from the collection.

The sorting parameters passed separated by a comma to the key:

sort=stats.ts_delay,-stats.bitrate

By default, ascending-order sort is used. To change the sorting direction, specify - before the field name.

The absence of a prefix before the filtering fields means that we do not allow sort fields in objects in our API.

We added an implicit sorting everywhere in our API. For example, if sorting by the provider field (i. e. a non-unique field) is applied, then several groups of objects will be returned incomprehensibly sorted. To avoid such a situation, we add fields like name and position to the end of the sorting request, applying implicit sorting.

If they are specified explicitly in the list of sorting fields, then re-sorting by them is no longer performed.

We do not specify a list of specific fields for implicit sorting since it may change at any time if we decide to use it differently for a particular case.

Limiting collection rows (cursors)¶

If a collection is large, HTTP API should be able to return them part by part.

To get the first 100 rows (elements) of a collection, use the limit=100 parameter in the query string.

The question is how to retrieve the next 100 elements.

In response to the request for the first 100 elements, the next field (and in the next samples prev) is returned. This field value should be passed in the cursor= parameter in the query string to fetch the following:

$ curl -sS "http://FLUSSONIC-IP:8080/streamer/api/v3/streams?select=name&limit=1&name_like=a&sort=name"  |jq 
{
  "estimated_count": 5,
  "next": "JTI0cG9zaXRpb25fZ3Q9MiZuYW1lX2d0PWEx",
  "prev": null,
  "streams": [
    {
      "effective": {
        "name": "a1",
        "position": 2,
        "section": "stream",
        "static": true
      },
      "name": "a1"
    }
  ],
  "timing": {
    "filter": 0,
    "limit": 0,
    "load": 3,
    "select": 0,
    "sort": 0
  }
}

$ curl -sS "http://FLUSSONIC-IP:8080/streamer/api/v3/streams?select=name&limit=1&name_like=a&sort=name&cursor=JTI0cG9zaXRpb25fZ3Q9MiZuYW1lX2d0PWEx"  |jq 
{
  "estimated_count": 5,
  "next": "JTI0cG9zaXRpb25fZ3Q9MyZuYW1lX2d0PWEy",
  "prev": "JTI0cG9zaXRpb25fbHQ9MyZuYW1lX2x0PWEyJiUyNHJldmVyc2VkPXRydWU=",
  "streams": [
    {
      "effective": {
        "name": "a2",
        "position": 3,
        "section": "stream",
        "static": true
      },
      "name": "a2"
    }
  ],
  "timing": {
    "filter": 0,
    "limit": 0,
    "load": 1,
    "select": 0,
    "sort": 0
  }
}

Using consecutive queries, we get all the elements of the data selection. The cursor is arranged quite simple: the values of the sorting fields for the last element of the selection are encoded in it, and an additional filtering is performed before the data is returned.

Limiting the field set of the result¶

We provide the possibility to return only a limited set of fields. The total number of fields in the stream data of the media server exceeds 100, and if you need to get only the stream names, then it is rather impractical to request all of the information.

To return only a limited set of fields, specify the select option, for example, select=name,title. You can also request the fields of nested objects: select=name,status.media_info.

Creating and updating (upsert)¶

The REST concept dictates the distinction between creating and updating methods for already existing objects.

We have practically abandoned this separation and prefer a more robust option with simultaneous creation or updating of objects. If the methods for creating and updating are separated, the executed code has to have a condition expression with repetitions and perform a loop check. We made it so that, firstly, an object is created on the server. If we get a response that such an object already exists, then try to update it (or not), depending on the situation.

In fact, this concept corresponds to JSON merge patch document format processing rules:

• If the provided merge patch contains JSON members that do not appear within the target, those members are added.
• If the target does contain the member, the value is replaced.
• Null null values in the merge patch are given special meaning to indicate the removal of existing values in the target.
• If the patch is anything other than an object, the result will always be to replace the entire target with the entire patch.
• It's not possible to patch part of a target that is not an object, such as to replace just some of the values in an array.

Make a PUT request:

curl -X PUT -H "Content-Type: application/json" -d '{"title":"ORT"}' "http://FLUSSONIC-IP:8080/streamer/api/v3/streams/ort"

It is very convenient for the API that the object ID occupies only one segment. It means that if, for example, the stream name is a compound name (like sports/football) then you should escape the / character using %2F to access it over API:

curl -X PUT -H "Content-Type: application/json" -d '{"key":"value"}' "http://FLUSSONIC-IP:8080/streamer/api/v3/streams/sports%2Ffootball"

We call this approach UPSERT because it is similar to SQL UPSERT.

Idempotency token for POST¶

The idempotency token guarantees that the same operation will not be executed twice. It is especially important in a cloud platform to make sure that credits are not deducted from the account several times for the same action.

The example of idempotency token usage procedure:

1. A client uses POST request for stream creation. Each client's request includes the Idempotency-Key header. This is a unique value generated by the client which the server uses to recognize subsequent retries of the same request.
2. When a client creates a new stream in the cloud, the name of the stream is generated by Flussonic (not on the client’s side) – this guarantees the stream name uniqueness.
3. So if there is some network interruption, the client has to retry the request with the same Idempotency-Key. Flussonic will recognize that it is the same request and return the stream name generated for it. On the other hand, if there is no Idempotency-Key in the request, Flussonic will consider it as a separate request and generate a new stream name.

Reading objects¶

To read an object use a GET HTTP method:

curl -sS "http://FLUSSONIC-IP:8080/streamer/api/v3/streams/ort"  |jq 

Response: JSON-encoded file with the information about the object.

{
  "effective": {
    "name": "ort",
    "position": 7,
    "section": "stream",
    "static": true
  },
  "name": "ort",
  "named_by": "config",
  "position": 7,
  "static": true,
  "stats": {
    "alive": false,
    "bytes_in": 0,
    "bytes_out": 0,
    "client_count": 0,
    "dvr_enabled": false,
    "dvr_only": false,
    "dvr_replication_running": false,
    "id": "61496e6e-a22f-46af-bce5-5479f9067ead",
    "input_error_rate": 0,
    "last_access_at": 1632202350648,
    "lifetime": 0,
    "opened_at": 1632202350648,
    "out_bandwidth": 0,
    "publish_enabled": false,
    "remote": false,
    "retry_count": 19,
    "running": true,
    "running_transcoder": false,
    "start_running_at": 1632202350648,
    "transcoder_overloaded": false
  }
}

Removing objects¶

To delete an object, use a DELETE HTTP method:

curl -sS -X DELETE "http://FLUSSONIC-IP:8080/streamer/api/v3/streams/ort"

Response: HTTP 204 with empty body response.