Image and Video Coding: Video Coding Standards s k [ x , y ] u k [ x - PowerPoint PPT Presentation

Image and Video Coding: Video Coding Standards s k [ x , y ] u k [ x , y ] quantization indexes q k [ x , y ] transform & entropy bitstream quantization coding − scaling & input pictures inv. transform u ′ k [ x , y ] s k [ x , y ] ˆ s ∗ k [ x , y ] intra mode buffer for decision current intra prediction modes picture s ∗ k [ x , y ] intra-picture f iprd ( s ∗ k [ x , y ] ) prediction coding mode in-loop decision filtering motion-comp. s ′ ref [ x + m x , y + m y ] prediction s ′ k [ x , y ] motion data decoded motion picture estimation buffer coding modes, intra prediction modes, motion data

Video Coding Standards / Scope of Standards Video Coding Standards input pre- output bitstream post- video video processing processing encoder decoder samples samples scope of standards Why Standards ? Bitstream generated by an encoder should be reliably decodable by decoders of other manufacturers Image and video coding standards define “interface” between encoder and decoder Freedom for manufacturers to adapt actual implementations Scope of Video Coding Standards 1 Bitstream syntax (including constraints for transmitted and derived parameters) 2 Decoding process (example decoding process for conforming bitstreams) Encoding, pre- and post-processing are out of the scope of standards No guarantee of image/video quality Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 2 / 35

Video Coding Standards / Scope of Standards Bitstream Syntax Coded Representation Samples are representated by coding parameters Coding modes, motion parameters, intra prediction modes Quantization indexes ... Bitstream Syntax Format for representating coding parameters in the bitstream Order of transmission Conditions for presence Motion data are only included for inter-picture coded blocks Intra prediction modes are only included for intra-picture coded blocks Entropy coding Codeword table Binarization and probability models Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 3 / 35

Video Coding Standards / Scope of Standards Example for Bitstream Syntax: Motion Data in H.265 | HEVC Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 4 / 35

Video Coding Standards / Conformance Points Interoperability and Conformance Conformance Points Standards are designed for broad range of potential application areas Some features may not be appropriate for all application scenarios (would unnecessarily increase complexity of decoder implementations) Define different conformance points using profiles & levels Interoperability between applications with similar requirements Profiles and Levels Profile : Subset of coding tools supported in standards Level : Maximum values for key parameters (picture size, frame rate, bit rate, buffer sizes) Conformance Conforming bitstream : Obeys all constraints on syntax elements and derived parameters Conforming encoder : Generates conforming bitstreams Conforming decoder : Produces same result as decoding process specified in standard Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 5 / 35

Syntax Features and Coding Tools / General Design Syntax Features and Coding Tools Development of Video Coding Standards Videos have by far the most data of all information types we use today Usage of videos as well as resolutions continuously increase Always desire higher compression capabilities Advances in computing devices enables new possibilities in video compression Research in video compression offers new directions Standard include well-tested features implementable for considered applications Standards included in Comparison H.262 | MPEG-2 Video (first version: 1995) H.264 | AVC: Advanced Video Coding (first version: 2003) H.265 | HEVC: High Efficiency Video Coding (first version: 2013) H.266 | VVC: Versatile Video Coding (first version: 2020) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 6 / 35

Syntax Features and Coding Tools / General Design Basic Design: Block-Based Hybrid Video Coding s k [ x , y ] u k [ x , y ] quantization indexes q k [ x , y ] entropy bitstream transform & quantization coding − Hybrid Video Coding scaling & All standards follow same basic design input pictures inv. transform Differ in many details u ′ k [ x , y ] s k [ x , y ] ˆ s ∗ k [ x , y ] Analyze Features in Following Areas intra mode Supported temporal coding structures buffer for decision current intra prediction modes Partitioning into blocks picture Intra-picture prediction s ∗ k [ x , y ] intra-picture f iprd ( s ∗ k [ x , y ] ) prediction Transform and quantization coding mode in-loop filtering decision Entropy coding motion-comp. s ′ ref [ x + m x , y + m y ] prediction s ′ k [ x , y ] Motion-compensated prediction Motion parameter coding motion data decoded motion picture estimation In-loop filtering buffer coding modes, intra prediction modes, motion data Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 7 / 35

Syntax Features and Coding Tools / General Design Basic Design: Block-Based Hybrid Video Coding s k [ x , y ] u k [ x , y ] quantization indexes q k [ x , y ] entropy bitstream transform & quantization coding − Hybrid Video Coding scaling & All standards follow same basic design input pictures inv. transform Differ in many details u ′ k [ x , y ] s k [ x , y ] ˆ s ∗ k [ x , y ] Analyze Features in Following Areas intra mode Supported temporal coding structures buffer for decision current intra prediction modes Partitioning into blocks picture Intra-picture prediction s ∗ k [ x , y ] intra-picture f iprd ( s ∗ k [ x , y ] ) prediction Transform and quantization coding mode in-loop filtering decision Entropy coding motion-comp. s ′ ref [ x + m x , y + m y ] prediction s ′ k [ x , y ] Motion-compensated prediction Motion parameter coding motion data decoded motion picture estimation In-loop filtering buffer coding modes, intra prediction modes, motion data Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 7 / 35

Syntax Features and Coding Tools / Pictures and Partitioning Picture Types and Supported Temporal Coding Structures H.262 | MPEG-2 Video Three pictures types: I, P, and B I P P P P P P P P No support of multiple reference pictures Two types of possible coding structures: IPPP: Only possibility for low delay (no bi-prediction) I/P B B B P B B B P IBBP: B picture as non-reference picture (between two I/P) H.264 | AVC, H.265 | HEVC, and H.266 | VVC I, P, and B slices (picture can consist of multiple slices) Decoupling of coding order and picture/slice types I/B B B B B B B B B B pictures can be used as reference picture More advanced and more efficient coding structures B picture and multiple reference pictures for low delay Hierarchical B pictures when high delay is possible I/B B B B B B B B B Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 8 / 35

Syntax Features and Coding Tools / Pictures and Partitioning Basic Concepts for Picture Partitioning into Blocks H.262 | MPEG-2 Video quadtree Fixed 16 × 16 macroblocks Coding mode (intra/inter) selected on macroblock basis H.264 | AVC 16 × 16 macroblocks with potential splitting Coding mode selected on macroblock basis multi-type tree H.265 | HEVC Quadtree (QT) partitioning of CTUs (up to 64 × 64) Coding mode selected on CU basis (result of QT partitioning) H.266 | VVC Multi-type tree (MTT) partitioning of CTUs (up to 128 × 128) Coding mode selected on CU basis (result of MTT partitioning) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 9 / 35

Syntax Features and Coding Tools / Pictures and Partitioning Partitioning for Prediction and Transform Coding Blocks for Motion-Compensated Prediction N × ( N / 2 ) ( N / 2 ) × N ( N / 2 ) × ( N / 2 ) N × N MPEG-2: 16 × 16 macroblocks H.264 | AVC: Sub-partitioning of macroblocks (4 × 4 to 16 × 16 blocks) H.265 | HEVC: Sub-partitioning of coding units N × ( N / 4 ) N × ( 3 N / 4 ) ( N / 4 ) × N ( 3 N / 4 ) × N (4 × 8/8 × 4 to 64 × 64 blocks) H.266 | VVC: Coding units (result of MTT) (4 × 8/8 × 4 to 128 × 128 blocks) Blocks for Intra Prediction and Transform Coding CTU MPEG-2: 8 × 8 transform blocks H.264 | AVC: 4 × 4 or 8 × 8 (also 16 × 16 for intra) H.265 | HEVC: Additional QT partitioning of CUs H.266 | VVC: Coding units (result of MTT), intra sub-partitioning Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 10 / 35

Syntax Features and Coding Tools / Intra-Picture Prediction Intra-Picture Prediction H.262 | MPEG-2 Video DC prediction only (quantization index) H.264 | AVC 9 spatial intra prediction modes (8 angles) H.265 | HEVC 35 spatial intra prediction modes (33 angles) -32 0 32 -32 − 135 ◦ H.266 | VVC 67 conventional prediction modes (out of 93, wide angles) Position dependent prediction sample filtering (PDPC) 0 Multi-reference line intra prediction (MRL) Matrix-based intra prediction (MIP) Cross-component linear model (CCLM) for chroma 45 ◦ 32 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Standards 11 / 35