Decoding the Video Codec Wars: H.264, HEVC, and AV1 Compared for Streaming

The cost of streaming heavily depends on two processes: compression and decompression of video data. And it’s the video codec that unifies them.

Many companies have tried to create the perfect codec. In the 2000s and 2010s, this led to a market war that affected everyone: viewers, content distributors, and equipment developers. The solutions we use today are the true winners in the competition for video stream quality and visual excellence.

Today, three codecs lead the way: H.264, HEVC, and AV1. In this article, we will explore their history and understand the differences and benefits for TV/IPTV providers, OTT services, and professional production companies.

What is a Video Codec

A video codec (compressor-decompressor) is a technology used to encode and decode video files. Encoding compresses the video to reduce file size, making it easier to transmit and store. Decoding restores the compressed video to its original quality for playback. The choice of codec affects video quality, file size, and compatibility with various devices.

It’s crucial to distinguish between video codecs and container formats:

Codec: This is the software that compresses and decompresses video files. For example, H.264 and HEVC (H.265) are codecs that determine how video data is encoded to reduce file size and decoded for playback.

Container Format: This is the file format you see, like .avi, .mov, or .mp4. Containers bundle video and audio streams into a single file but do not handle compression themselves. They act as the wrapper for codec-encoded content.

Example: A video file with an .mp4 extension might use the H.264 codec for video compression and AAC for audio compression. The .mp4 container simply holds these streams together. This capability is crucial for efficient video transcoding in today’s streaming environments.

How does a codec work

Encoded video frames consist of a matrix of pixels. A video codec identifies and tracks groups of similar pixels with identical characteristics (such as background color). Instead of recording each pixel’s details (like color and brightness) individually, the codec stores only the key pixel and a count of how many times this pixel is repeated until its color changes. The codec determines image quality by setting a threshold where minor differences between pixels are considered identical.

This approach is used by simpler video codecs. More advanced codecs incorporate additional factors, such as the movement of pixel groups that form the image, segmentation with varying compression quality, and the use of sequences of frames encoded in different ways and presented in a specific order.

Modern codecs are designed to exploit the psychophysical characteristics of human vision and the nervous system, enabling them to significantly reduce data size without sacrificing image quality.

Codec Wars: A Quest for the Perfect Video Codec

In the early 2000s, as the video streaming industry expanded rapidly, companies and organizations involved in codec development aimed to shape the future of video technology. This competition, known as the Codec Wars, was not only about developing better codecs but also about controlling market standards and technology. Key players like codec developers and tech giants invested heavily in research, development, and lobbying to influence industry standards, pricing, and technology control.

Companies strive to control codec technologies to influence pricing and market adoption. Licensing fees and royalties for codecs like HEVC have influenced decisions in the industry. In contrast, royalty-free codecs like AV1 aim to disrupt this dynamic by reducing costs.

H.264 (AVC)

Released in 2003, H.264, or AVC (Advanced Video Coding), quickly became the standard for video compression. Developed by the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group, H.264 offered an efficient balance between video quality and file size. It introduced advancements such as better compression efficiency and support for a range of resolutions and frame rates.

YouTube initially adopted H.264 for its video streaming services. This choice allowed users to stream videos smoothly across various devices, balancing quality and file size effectively. The flexibility of H.264 also made it a popular choice for use in IPTV encoder setups and other streaming technologies.

How H.264 (AVC) works

• Block-Based Motion Compensation: codec divides a video frame into small blocks (usually 16x16 pixels) and tracks how these blocks move from frame to frame. By only recording changes between blocks, H.264 compresses video more efficiently.

• Variable Block Size: Instead of using a fixed block size, H.264 allows different sizes for blocks, such as 4x4 or 8x8 pixels. This flexibility helps compress videos better, especially in scenes with varying motion.

• Intra-Frame Prediction: For areas of a video that don’t change, H.264 uses information from neighboring pixels to predict pixel values, which reduces the amount of data needed.

• Entropy Coding: H.264 uses two methods, CAVLC and CABAC, to compress data further. CABAC is more efficient but requires more processing power.

Why H.264 Was Important:

• Efficiency: Offered significant improvement in compression efficiency compared to older codecs like MPEG-2, enabling high-quality video at lower bit rates.

• Widespread Adoption: Its efficiency and relatively straightforward licensing led to extensive adoption across platforms and devices.

Enter HEVC: The Next Big Step

As video quality demands grew with the rise of HD and 4K content, the need for better compression became evident. This led to the development of HEVC (H.265), finalized in 2013 as the successor to H.264. Developed by the ISO/IEC Moving Picture Experts Group and the ITU-T Video Coding Experts Group, HEVC aimed to double the data compression ratio compared to H.264.

Netflix adopted HEVC to deliver 4K streaming content. This transition enabled Netflix to provide high-quality video while managing bandwidth efficiently, offering superior picture quality to its subscribers.

How HEVC (H.265) works

HEVC (H.265) introduced several advancements over H.264:

• Coding Tree Units (CTUs): Instead of small blocks, HEVC uses larger blocks (up to 64x64 pixels). This larger block size improves compression, especially for high-resolution videos like 4K, by reducing the overhead associated with smaller blocks.

• Advanced Motion Prediction: HEVC enhances motion tracking with more precise methods. It uses smaller movements and multiple reference frames to better predict object motion, improving data efficiency.

• More Prediction Modes: HEVC provides a broader range of prediction techniques to better handle different parts of a frame, leading to improved compression accuracy.

• Improved Entropy Coding: HEVC employs an advanced version of CABAC, offering better compression but with increased complexity compared to H.264.

• Parallel Processing: HEVC supports dividing a video into smaller sections for simultaneous processing, which speeds up encoding and decoding, particularly for high-resolution content.

What HEVC aimed to achieve

• Improved Compression: Promised to double the compression efficiency of H.264, delivering the same video quality at half the bit rate or better quality at the same bit rate, crucial for handling high-definition and 4K content.

• Better Quality for Emerging Technologies: Addressed the growing demand for 4K video by handling large video files more efficiently.

Challenges with HEVC:

• Complexity: Introduced more advanced compression techniques, increasing encoding and decoding complexity.

• Licensing Costs: Encumbered by multiple patent pools, leading to significant licensing fees and barriers to widespread adoption.

The Rise of AV1: a new contender

As the industry evolved, there was a push for a codec offering better performance without the high costs associated with HEVC. This led to the development of AV1. Finalized in 2018 by the Alliance for Open Media (AOMedia), AV1 was designed as a royalty-free alternative to HEVC, aiming to improve compression efficiency and reduce licensing costs.

YouTube has started using AV1 for some of its video streams, particularly for high-resolution and HDR content. This adoption allows YouTube to deliver improved video quality at lower bit rates, benefiting users with slower internet connections. AV1’s adoption is also gradually increasing in various streaming encoder technologies.

How AV1 works

AV1 aims to offer better performance than HEVC:

• Block-Based and Tile-Based Coding: AV1 uses a mix of block-based and tile-based coding. It can handle blocks ranging from 4x4 to 128x128 pixels, allowing it to adapt to different video content efficiently and compress it effectively.

• Adaptive Quantization: AV1 adjusts the level of detail it keeps based on the complexity of the video. This dynamic adjustment helps compress video more effectively by not wasting data on simpler scenes.

• Advanced Prediction Techniques: AV1 includes new and improved methods to predict video content, which helps reduce the data needed for complex scenes.

• Multi-Layered Entropy Coding: AV1’s entropy coding is more sophisticated, using new techniques to compress data even more efficiently.

• HDR Support: AV1 handles high dynamic range (HDR) content better, improving color accuracy and quality for modern displays.

What AV1 aimed to achieve

• Superior Compression: Designed to surpass HEVC in compression efficiency, offering approximately 30% better compression and higher quality video at lower bit rates.

• Cost Efficiency: Eliminates licensing fees and patent-related costs, making AV1 a more affordable option.

• Future-Proofing: Includes features to handle not only 4K but also 8K video and beyond, preparing for future advancements.

Challenges with AV1

• Complexity: Advanced features result in higher encoding and decoding complexity, requiring more computational power and advanced hardware.

• Adoption: While growing, AV1’s adoption is still limited compared to H.264 and HEVC, due to slower rollout of supporting hardware and software.