Image and Video Coding: Hybrid Video Coding s n 1 [ x , y ] s n [ - PowerPoint PPT Presentation

Image and Video Coding: Hybrid Video Coding s ′ n − 1 [ x , y ] s n [ x , y ] m k = ( m x , m y ) k x y

Hybrid Video Coding Last Lectures: Block-Based Image Coding s k [ x , y ] u k [ x , y ] t k [ x , y ] q k [ x , y ] bitstream scalar entropy 2D block quantization coding transform − scaling & input picture inv. transform u ′ k [ x , y ] ˆ s k [ x , y ] s ′ k [ x , y ] buffer for intra-picture current prediction picture 1 Transform Coding of Sample Blocks Transform: Energy compaction / decorrelation Quantization: Approximation of input signal Entropy Coding: Lossless compression of quantization indexes 2 Intra-Picture Prediction between Transform Blocks Utilization of statistical dependencies between transform blocks Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 2 / 42

Hybrid Video Coding / Frame Difference Coding Similarities between Successive Pictures in a Video Sequence t Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 3 / 42

Hybrid Video Coding / Frame Difference Coding General Idea of Hybrid Video Coding “It has been customary in the past to transmit successive complete images of the transmitted picture. [ · · · ] In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.” Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 4 / 42

Hybrid Video Coding / Frame Difference Coding Simple Variant: Frame Difference Coding rec. previous picture s ′ prev [ x , y ] current original picture s [ x , y ] prediction error u [ x , y ] Two coding modes for each block ( transmit flag per block) 1 Intra prediction: ˆ s [ x , y ] = f ipred ( s ′ [ x , y ] ) (use neighboring reconstructed samples) 2 Inter prediction: ˆ s [ x , y ] = s ′ prev [ x , y ] (copy co-located samples of rec. previous picture) Transform coding of prediction error Get prediction error: u [ x , y ] = s [ x , y ] − ˆ s [ x , y ] Transform coding: u [ x , y ] �→ u ′ [ x , y ] ( transmit quantization indexes { q k } ) s ′ [ x , y ] = ˆ s [ x , y ] + u ′ [ x , y ] Reconstruction: Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 5 / 42

Hybrid Video Coding / Frame Difference Coding Frame Difference Coding: Inter-Picture Prediction Error Johnny (original) frame difference (42.6 dB) video properties static background low object motion few spatial details small inter prediction error BQTerrace (original) frame difference (27.7 dB) video properties moving camera moderate object motion lots of spatial details large inter prediction error Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 6 / 42

Hybrid Video Coding / Frame Difference Coding Coding Efficiency of Frame Difference Coding Johnny (1280 × 720, 60 Hz) BQTerrace (1920 × 1080, 60 Hz) 44 38 43 37 42 36 41 PSNR (Y) [dB] PSNR (Y) [dB] 35 40 FD 39 34 Intra 38 33 FD 37 Intra 32 36 31 35 34 30 0 2 4 6 8 10 0 10 20 30 40 50 60 70 80 bit rate [Mbit/s] bit rate [Mbit/s] Large gains for video conferencing content (static background, low motion) Comparably small gains for videos with strong camera or object motion Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 7 / 42

Hybrid Video Coding / Motion-Compensated Prediction Motion-Compensated Prediction reconstructed reference picture current picture displacement vector for current block best matching m = ( m x , m y ) block in reference picture x moving object displaced object y General Idea for Improving the Inter-Picture Prediction Predict a current block using a displaced block in an already coded picture s [ x , y ] = s ′ ˆ ref [ x + m x , y + m y ] Transmit displacement vectors m = ( m x , m y ) as part of the bitstream Encoder has to select suitable displacement vector m = ( m x , m y ) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 8 / 42

Hybrid Video Coding / Motion-Compensated Prediction Estimation of Displacement Vector / Motion Vector Straightforward Motion Search rec. reference picture s ′ ref [ x , y ] � � � � Define search range R = − m max ; m max × − m max ; m max Test all vectors m = ( m x , m y ) in search range R Calculate cost measure C ( m ) Compare to current minimum C min Choose vector m that minimizes cost measure C ( m ) current original picture s [ x , y ] Typical Cost Measures for Motion Search Sum of absolute differences (SAD) � � � � s [ x , y ] − s ′ D SAD ( m x , m y ) = ref [ x + m x , y + m y ] � � � x , y Lagrangian cost of SAD and bit rate for motion vector J ( m x , m y ) = D SAD ( m x , m y ) + λ · R ( m x , m y ) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 9 / 42

Hybrid Video Coding / Motion-Compensated Prediction Illustration of Motion-Compensated Prediction rec. reference picture s ′ ref [ x , y ] original current picture s [ x , y ] motion vector field m [ x , y ] referenced blocks in s ′ ref [ x , y ] prediction ˆ s [ x , y ] for current picture pred. error u [ x , y ] = s [ x , y ] − ˆ s [ x , y ] Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 10 / 42

Hybrid Video Coding / Motion-Compensated Prediction Motion-Compensated Prediction: Inter-Picture Prediction Error Johnny (original) frame difference (42.6 dB) MCP error (44.6 dB) BQTerrace (original) frame difference (27.7 dB) MCP error (33.7 dB) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 11 / 42

Hybrid Video Coding / Motion-Compensated Prediction Do We Still Need Intra-Picture Coding Modes ? rec. reference picture s ′ ref [ x , y ] current original picture s [ x , y ] prediction error u [ x , y ] Not all regions can be well predicted by motion-compensated prediction Uncovered background New regions entering the image area (due to object or camera motion) Motion that cannot be well represented by a block-wise translational motion field Unmatched prediction can decrease coding efficiency Prediction error can contain more high-frequency signal energy than original signal Direct coding or intra-picture prediction can lead to better coding efficiency Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 12 / 42

Hybrid Video Coding / Motion-Compensated Prediction Coding Efficiency Impact of Intra-Predicted Blocks Basketball Drive, IPPP Basketball Drive, IPPP 38 30 all coding modes 37 25 bit-rate increase due to bit-rate increase [%] disabling of intra-picture PSNR (Y) [dB] coding modes in inter pictures 36 20 (on average, 11%) 35 15 intra-picture coding modes are disabled 34 10 in inter pictures 33 5 32 0 0 2 4 6 8 10 32 33 34 35 36 37 38 bit rate [Mbit/s] PSNR (Y) [dB] Intra coding modes are important for areas that cannot be well predicted by MCP (uncovered background, regions with complicated non-translational motion, ...) Enabling of intra coding modes increases coding efficiency Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 13 / 42

Hybrid Video Coding / Block-based Hybrid Video Coding Hybrid Video Coding rec. reference picture s ′ ref [ x , y ] current original picture s [ x , y ] prediction error u [ x , y ] Two coding modes for each block ( transmit coding mode) 1 Intra prediction: ˆ s [ x , y ] = f ( s ′ [ x , y ] , p intra ) ( transmit intra prediction mode p intra ) 2 Inter prediction: ˆ s [ x , y ] = s ′ ref [ x + m x , y + m y ] ( transmit motion vector m = ( m x , m y ) ) Transform coding of prediction error Prediction error: u [ x , y ] = s [ x , y ] − ˆ s [ x , y ] Transform coding: u [ x , y ] �→ u ′ [ x , y ] ( transmit quantization indexes { q k } ) Reconstruction: s ′ [ x , y ] = ˆ s [ x , y ] + u ′ [ x , y ] Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 14 / 42

Hybrid Video Coding / Block-based Hybrid Video Coding Block-Based Image Encoder with Intra-Picture Prediction s k [ x , y ] u k [ x , y ] quantization indexes q k [ x , y ] entropy bitstream transform & quantization coding − scaling & input picture inv. transform u ′ k [ x , y ] s k [ x , y ] = f iprd ( s ∗ ˆ k [ x , y ] ) s ∗ k [ x , y ] intra mode buffer for decision current intra prediction modes picture intra-picture f iprd ( s ∗ k [ x , y ] ) prediction Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 15 / 42