Image and Video Coding: Motion Estimation and Coding 4 5 6 B C - PowerPoint PPT Presentation

Image and Video Coding: Motion Estimation and Coding 4 5 6 B C D 1 D 0 3 7 A current 2 block 1 E 1 E 0

Last Lecture Last Lecture: Hybrid Video Coding Hybrid Video Coding Partitioning of pictures into blocks current picture displacement vector for current block Block-adaptive prediction m = ( m x , m y ) s [ x , y ] = f ( s ′ [ x , y ] , p intra ) intra: ˆ x s [ x , y ] = s ′ inter: ˆ ref [ x + m x , y + m y ] Transform coding of prediction error displaced object y u [ x , y ] = s [ x , y ] − ˆ s [ x , y ] reconstructed reference picture Motion-Compensated Prediction best matching block in Key technique for utilizing temporal dependencies reference picture Parameters that impact coding efficiency Accuracy of motion vectors moving object Choice of interpolation filters ... Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 2 / 38

Picture Partitioning / Variable Block Sizes Picture Partitioning for Motion-Compensated Prediction Effectiveness of Motion-Compensated Prediction Large blocks: Rather inaccurate motion representation, but small rate overhead for motion data Small blocks: More accurate prediction signal, but higher motion data rate Transform Coding of Prediction Error Signal Avoid transform over motion edges (edges in residual signal decrease efficiency of transform coding) Larger blocks for prediction provide more flexibility for transform partitioning Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 3 / 38

Picture Partitioning / Variable Block Sizes Variable Block Sizes for Motion-Compensated Prediction Regions for Motion Description Moving objects can have arbitrary shapes Variable size regions for motion description Partitioning has to be signaled in bitstream Sample-accurate partitionings are too expensive (and difficult to decide in an encoder) Need simple, but effective solutions Blocks of variable sizes Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 4 / 38

Picture Partitioning / Variable Block Sizes Simple Partitioning Concepts with Variable Block Sizes Most Common Adaptive Partitioning Initial partitioning into fixed size blocks Signal-adaptive subdivision of initial blocks Quadtree Partitioning Consecutive splits into four sub-blocks Allows signal-adaptive subdivision Low signaling overhead (one flag per node) root 1 Extended Tree Partitioning Multiple split types per node 0 1 0 1 (e.g., quadtree & binary splits) 0 0 0 1 0 1 1 0 Additional flexibility for subdivision Results in larger blocks (on average) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 5 / 38

Picture Partitioning / Variable Block Sizes Motion-Compensated Prediction for Chroma Multiple Color Components luma and chroma interpolation filters of HEVC Same motion in luma and chroma components phase -3 -2 -1 0 1 2 3 4 1/4 -1 4 -10 58 17 -5 1 Use same partitioning & same motion vectors luma 1/2 -1 4 -11 40 40 -11 4 -1 3/4 1 -5 17 58 -10 4 -1 Motion Compensation in YCbCr 4:2:0 1/8 -2 58 10 -2 1/4 -4 54 16 -2 Chroma blocks have quarter number of samples 3/8 -6 46 28 -4 Chroma motion vectors have double precision chroma 1/2 -4 36 36 -4 5/8 -4 28 46 -6 Twice as many phases as in luma 3/4 -2 16 54 -4 7/8 -2 10 58 -2 Typically: Simpler interpolation filters Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 6 / 38

Picture Partitioning / Supported Block Sizes in Video Coding Standards Picture Partitioning in Early Video Coding Standards MPEG-2 Video 16 × 16 macroblocks (decision: intra or inter) Y One motion vector per macroblock (all components) Transform blocks of 8 × 8 samples Cb Macroblock in 4:2:0 format: 6 transform blocks Cr No adaptivity in partitioning H.263 and MPEG-4 Visual 16 × 16 macroblocks and 8 × 8 transform blocks Inter-16x16 Inter-8x8 Two inter coding mode per macroblock MV 0 MV 1 1 One motion vector per macroblock MV 0 2 Macroblock is split into four 8 × 8 blocks, MV 2 MV 3 with one motion vector per 8 × 8 block One level of quadtree partitioning Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 7 / 38

Picture Partitioning / Supported Block Sizes in Video Coding Standards Block Partitioning in H.264 | AVC Picture Partitioning Inter-16x16 Inter-16x8 Inter-8x16 Inter-8x8 16 × 16 macroblocks (decision: intra or inter) Motion Partitioning Two levels of macroblock partitioning 1 Four modes for partitioning into sub-macroblocks Inter-16x16, Inter-16x8, Inter-8x16, Inter-8x8 2 8 × 8 sub-macroblocks can be further subdivided 8x8, 8x4, 4x8, 4x4 8x8 8x4 4x8 4x4 Transform Partitioning (High profile) Two modes for partitioning of luma residual luma 8x8 luma 4x4 chroma 1 Four 8 × 8 transform blocks not allowed if motion blocks smaller 8 × 8 are used 2 Sixteen 4 × 4 transform blocks Chroma: Only 4 × 4 transforms Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 8 / 38

Picture Partitioning / Supported Block Sizes in Video Coding Standards Block Partitioning in H.265 | HEVC Partitioning into Coding Units CTU 1 Initial partitioning into Coding Tree Units (CTUs) Fixed size of 64 × 64 (typical), 32 × 32, or 16 × 16 2 Quadtree partitioning into Coding Units (CUs) Up to minimum CU size, with ( N × N ) min ≥ 8 × 8 Coding Unit : Decision between intra and inter Motion Description (inter coded CUs) Split CU into up to four Prediction Units (PUs) N × ( N / 2 ) ( N / 2 ) × N ( N / 2 ) × ( N / 2 ) N × N Supports non-square block shapes only if N > 8 , Prediction Unit : Coding of motion data N = N min Transform Coding N × ( N / 4 ) N × ( 3 N / 4 ) ( N / 4 ) × N ( 3 N / 4 ) × N Quadtree part. of CU into Transform Units (TUs) only if Minimum supported transform size: 4 × 4 N > 8 Transform Unit : Block sizes for transform coding Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 9 / 38

Picture Partitioning / Supported Block Sizes in Video Coding Standards Block Partitioning in H.266 | VVC Initial Partitioning no Bh Th Fixed size Coding Tree Units (CTUs) 1 1 1 1 / 2 1 / 2 4 2 4 CTU size of 128 × 128, 64 × 64, or 32 × 32 Q Bv Tv 1 / 4 Partitioning into Coding Units (CUs) 1 / 2 1 / 2 1 / 2 Multi-type tree partitioning of CTUs 1 / 2 1 / 2 1 / 4 Split types: quad, binary, ternary splits Some restrictions on multi-type tree split codeword No quad split after binary/ternary split no 0 Q 1 1 CU width and height: Multiple of 4 CTU Bh 1 0 0 1 Bv 1 0 1 1 Avoid redundant split sequences Q Th 1 0 0 0 Tc 1 0 1 0 Motion Description and Transform Coding Q Bh Tv Bv Motion data are signaled per CU Q Bv Tv Th Bh Th Bh One transform per color component of CU no no no no (exception: CU larger than max. transform) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 10 / 38

Picture Partitioning / Coding Order and Encoder Decision Coding Order of Blocks Transmission of Block Parameters 1 2 3 4 5 6 7 8 Map 2D arrangement of blocks into 1D bitstream 10 11 12 13 ... 9 Require defined coding order Coding Order of CTUs / Macroblocks Raster scan order From top to bottom, from left to right Coding Order of Blocks At each split node (z-scan): 1 From top to bottom 2 From left to right Properties for Prediction Blocks to the left and above always available Block above right is often available Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 11 / 38

Picture Partitioning / Coding Order and Encoder Decision Partitioning Selection in Encoder Lagrangian Mode Decision example : two-level quadtree Selection partitioning with minimum J = D + λ · R distortion D of reconstructed block number of bits R for sending all coding parameters Each node: Compare Lagrangian costs J noslpit for non-split block J split,A for split mode A (e.g., quad split) ... Decision Order 5 Need correct neighborhood for decisions (for predictors) Decide split modes in coding order 1 2 3 4 Evaluate tree in depth first order Rather complex (in particular for multi-type trees): Each sample is coded multiple times Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 12 / 38

Picture Partitioning / Coding Order and Encoder Decision Fast Partitioning Selection Approaches Fast Partitioning Algorithms example : two-level quadtree General Idea: Skip evaluation of unlikely partitionings Properties of typical video sequences Small inter blocks are rather unlikely Many inter blocks have zero prediction error Basic Fast Partitioning Design Evaluate non-split blocks first Don’t test splits if large block fulfills “quality criterion” 1 Block has zero residual signal, or All transform coefficients are less than threshold, or 2 4 6 7 ... Combination of top-down fast decisions with conventional depth-first Lagrangian mode decision 3 5 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Motion Estimation and Coding 13 / 38