Image and Video Coding: Improved Inter-Picture Prediction Review of - PowerPoint PPT Presentation

Image and Video Coding: Improved Inter-Picture Prediction

Review of Hybrid Video Coding Last Lectures: Hybrid Video Coding rec. reference picture s ′ ref [ x , y ] Hybrid Video Coding Partitioning of pictures into blocks Block-adaptive selection of prediction method 1 intra-picture prediction 2 motion-compensated prediction Transform coding of prediction error current original picture s [ x , y ] Effectiveness of Motion-Compensated Prediction Accuracy of motion vectors Choice of interpolation filters Prediction and coding of motion parameters Estimation algorithm for motion parameters ... Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 2 / 39

Advanced Motion-Compensated Prediction / Multiple Reference Pictures Multiple Reference Pictures s ′ s ′ s ′ s ′ k − 4 [ x , y ] k − 3 [ x , y ] k − 2 [ x , y ] k − 1 [ x , y ] s k [ x , y ] current r = 2 r = 1 r = 0 picture decoded picture buffer (DPB) Motion-Compensated Prediction with Multiple Reference Pictures Store multiple reconstructed pictures in a decoded picture buffer (DPB) Example: Sliding window buffer (FIFO), which contains the N most recently decoded pictures For each block: Select reference picture used for prediction Transmit reference picture index r in addition to motion vector Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 3 / 39

Advanced Motion-Compensated Prediction / Multiple Reference Pictures Encoder Control: Reference Picture Selection Typical Approach 1 Determine motion vector m r = ( m x , m y ) for each possible reference picture r ∈ [ 0 ; N − 1 ] 2 Select motion parameters { r , m r } among the pre-determined sets Criterion for Reference Picture Selection Lagrangian decision similar to motion search Choose motion parameters { r , m r } according to r ∗ = arg min ∀ r D ( r , m r ) + λ m · R ( r , m r ) D ( r , m r ) : Distortion between original block s [ x , y ] and prediction signal s ′ with r [ x + m x , y + m y ] R ( r , m r ) : Number of bits for reference index r and motion vector m r Usually: Same distortion measure as for sub-sample search (often SAD in Hadamard domain) Complexity Impact : Need N motion searches per block (instead of one) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 4 / 39

Advanced Motion-Compensated Prediction / Multiple Reference Pictures Coding Efficiency: Multiple Reference Pictures BQSquare (416 × 240, 60 Hz) Johnny (1280 × 720, 60 Hz) 60 25 bit-rate saving vs 1 ref. pic. [%] 8 ref. pics. (avg. 35.7%) bit-rate saving vs 1 ref. pic. [%] 8 ref. pics. (avg. 10.1%) 50 20 4 ref. pics. (avg. 29.0%) 4 ref. pics. (avg. 9.0%) 40 15 30 2 ref. pics. (avg. 5.3%) 10 20 5 10 2 ref. pics. (avg. 16.6%) 0 0 26 28 30 32 34 36 38 40 34 36 38 40 42 44 PSNR (Y) [dB] PSNR (Y) [dB] Coding Experiment (HEVC) : IPPP coding Use N = 1 , 2 , 4 , 8 preceeding pictures as reference picture (sliding window mechanism) Coding efficiency typically increases with the number of reference pictures Can have very large impact (depends on source material) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 5 / 39

Advanced Motion-Compensated Prediction / Multi-Hypothesis Prediction Multi-Hypothesis Prediction s ′ k − 4 [ x , y ] s ′ k − 3 [ x , y ] s ′ k − 2 [ x , y ] s ′ k − 1 [ x , y ] s k [ x , y ] current r = 2 r = 1 r = 0 picture decoded picture buffer (DPB) Motion-Compensated Prediction with Multiple Motion Hypotheses Weighted summation of multiple displaced reference blocks s ′ r k [ x + m k , x , y + m k , y ] Simplest and most commonly used variant (equal weigthing of individual prediction signals) K − 1 s [ x , y ] = 1 � s ′ ˆ r k [ x + m k , x , y + m k , y ] K k = 0 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 6 / 39

Advanced Motion-Compensated Prediction / Multi-Hypothesis Prediction Why Multi-Hypothesis Prediction ? Idealized Assumptions Individual prediction errors u k [ x , y ] = s [ x , y ] − s ′ r k [ x + m r , x , y + m r , y ] are uncorrelated All individual prediction errors have zero mean and the same variance σ 2 single Effect of Multi-Hypothesis Prediction Variance of multi-hypothesis prediction error for idealized assumptions  2  � K − 1 � K − 1 K − 1 1 1 1 � U 2 �   � � � � � σ 2 U 2 σ 2 multi = E = E  = = U k E k single K K 2 K 2  k = 0 k = 0 k = 0 1 σ 2 K · σ 2 multi = single Prediction error variance is reduced by factor K (compared to conventional prediction) Require less bit rate for achieving same distortion Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 7 / 39

Advanced Motion-Compensated Prediction / Multi-Hypothesis Prediction Block-Adaptive Multi-Hypothesis Prediction Coding Efficiency of Multi-Hypothesis Prediction Idealized assumptions: Prediction error variance is reduced by factor K 1 σ 2 K · σ 2 multi ≈ single In reality, reduction of prediction error variance is smaller (increase is also possible!) Bit rate for motion data (motion vectors and reference indexes) is increased Multi-hypothesis prediction does not improve coding efficiency for all blocks Block-Adaptive Selection of Number of Motion Hypotheses Select number of motion hypotheses for each block (signaled in bitstream) Video coding standards: Only up to two motion hypotheses 1 Uni-predicted blocks: Conventional motion-compensated prediction (one hypothesis) 2 Bi-predicted blocks: Averaging of two motion-compensated prediction signals (two hypotheses) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 8 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards Picture Types in Early Video Coding Standards I P P P P P P P P P I B B P B B P B B P 0 1 2 3 4 5 6 7 8 9 0 2 3 1 5 6 4 8 9 7 IPPP Coding I Pictures All pictures are coded in acquisition/display order all block are intra-coded Bi-prediction is not supported P Pictures intra or uni-prediction IBBP Coding last I/P as reference stored as reference picture One or more B pictures are inserted between two I/P pictures coded after P picture succeeding in display order B Pictures support bi-prediction using last two coded I/P pictures as references intra, uni-, or bi-prediction last two I/P as references never used as reference pictures not used as reference pic. P pictures use last I/P picture as reference picture Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 9 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards Bi-Prediction in Early Video Coding Standards forward backward prediction prediction bi-prediction I/P ( n ) B ( n + 2) B ( n + 3) I/P ( n + 1) Bi-Prediction in MPEG-2 Video, H.263, and MPEG-4 Visual B pictures support three inter-picture coding modes (selected on block basis) Forward prediction : Single-hypothesis prediction from preceding I/P picture Backward prediction : Single-hypothesis prediction from succeeding I/P picture Bi-directional prediction : Bi-prediction from preceding and succeeding I/P picture ! B pictures can only be coded as part of an BBP or BBI group Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 10 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards Picture Types in Modern Video Coding Standards I B B B P P P P B B I B B B P B B P P B 0 1 2 3 4 5 6 7 8 9 0 5 4 3 6 7 8 2 9 1 H.264 | AVC, H.265 | HEVC, and H.266 | VVC Generalized concept of bi-prediction and multiple reference pictures Arbitrary coding order of pictures (only limited by size of decoded picture buffer) Picture types (I, P, B) decoupled from coding order (actually: slice types instead of picture types) Reference Picture Lists P slices: One reference picture list (constructed from pictures in decoded picture buffer) B slices: Two reference picture lists (referred to as list 0 and list 1 ) Both reference picture list can include pictures from past and future Each stored picture (in DPB) can be included in either or both lists Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 11 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards Bi-Prediction in Modern Video Coding Standards current r 1 = 1 r 0 = 0 r 0 = 2 picture r 1 = 1 r 1 = 0 Inter-Picture Coding Modes in B Slices List 0 prediction : Uni-prediction with reference picture from list 0 List 1 prediction : Uni-prediction with reference picture from list 1 Bi-prediction with a picture from list 0 and a picture from list 1 Bi-prediction : Both motion hypotheses can use the same picture or different pictures Both motion hypotheses can stem from the same or different directions Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 12 / 39