Image and Video Coding: Intra Prediction & Picture Partitioning - PowerPoint PPT Presentation

Image and Video Coding: Intra Prediction & Picture Partitioning

Intra-Picture Prediction Transform Coding with Intra-Picture Prediction sample array original entropy bits 2d block quantization bitstream coding transform 8 × 8 block Transform Coding of Sample Blocks Decorrelating transform: Utilization of statistical dependencies (correlations) inside blocks There are also statistical dependencies between transform blocks Exploitation of these inter-block dependencies can increase coding efficiency example : averages of DC coefficients original picture 16 × 16 blocks (256 × 256) (DC coefficients) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 2 / 35

Intra-Picture Prediction / Prediction of Transform Coefficients Prediction of DC Coefficient (JPEG, MPEG-2 Video) Prediction of DC Quantization Index DC k − 1 DC k DC (direct current) represents scaled average of block samples block block (often similar to DC of neighboring blocks) k − 1 k Prediction using DC quantization index of block to the left Prediction difference DIFF = DC k − DC k − 1 is entropy coded DIFF = DC k − DC k − 1 Improves coding efficiency P ( DC − N 2 B − 1 / ∆) quantization: ∆ = 8 P ( DIFF ) σ = 37 . 0 σ = 5 . 29 H = 6 . 01 H = 3 . 10 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 3 / 35

Intra-Picture Prediction / Prediction of Transform Coefficients Example: Coding Efficiency Impact of DC Prediction ( 8 × 8 Blocks) Berlin (1200 x 900) 46 without prediction 44 with DC prediction 42 40 Luma PSNR [dB] 17 % 38 36 34 29 % 32 30 28 26 0 0.5 1 1.5 2 2.5 3 3.5 4 bits per luma sample Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 4 / 35

Intra-Picture Prediction / Prediction of Transform Coefficients Additional Dependencies between Transform Coefficients T T T T Mainly Horizontal Structures Mainly Vertical Structures Signal energy is concentrate in first Signal energy is concentrate in first column of transform coefficients row of transform coefficients Horizontal neighboring blocks have similar Vertical neighboring blocks have similar first column of transform coefficients first row of transform coefficients Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 5 / 35

Intra-Picture Prediction / Prediction of Transform Coefficients Advanced Intra Prediction in Transform Domain (H.263, MPEG-4 Visual) Prediction of transform coefficients t using reconstructed transform coefficients t ′ Quantization of prediction error ∆ t = t − ˆ t b t 0 , 0 = a ′ 0 , 0 + b ′ 0 , 0 ˆ ˆ t 0 , y = a ′ ∀ y : 2 0 , y ∀ x : t x , 0 = b ′ ˆ a x , 0 Three Intra Coding Modes 1 DC mode : Prediction of DC coefficient and zig-zag scan 2 Horizontal mode : Prediction of first column (from left block) and alternate-vertical scan 3 Vertical mode : Prediction of first row (from above block) and alternate-horizontal scan Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 6 / 35

Intra-Picture Prediction / Prediction of Transform Coefficients How to Select Suitable Coding Mode? Intra Mode Selection Syntax supports multiple intra prediction modes (mode is transmitted) Encoder has to select one of the supported modes Goal: Maximize coding efficiency Lagrangian Mode Decision 1 Transform original block of samples 2 Evaluate all supported coding modes (DC, horizontal, vertical) Perform prediction, quantization, and reconstruction of transform coefficient k − t k ) 2 Calculate SSD distortion D = � k ( t ′ Determine number of bits R for transmitting mode and block of quantization indexes 3 Select intra mode m that minimizes Lagrangian cost function J m = D m + λ · R m Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 7 / 35

Intra-Picture Prediction / Spatial Intra Prediction Intra Prediction: Transform Domain or Spatial Domain ? vertical prediction equivalent prediction simplified and improved in transform domain in spatial domain vertical prediction in spatial domain N s [ x , y ] = 1 � ˆ t [ x , 0 ] = t ′ s ′ [ x , − k ] above [ x , 0 ] ˆ s [ x , y ] = s ′ [ x , − 1 ] ˆ N k = 1 Intra Prediction in Transform Domain Disadvantage: Only possible if neighboring block is intra-coded (problem in video pictures) Intra Prediction in Spatial Domain Similar complexity than similar prediction in transform domain Usage of directly adjacent samples improves coding efficiency Main Advantages: Can also be applied if neighboring blocks are coded in an inter mode Straightforward extension to multiple prediction directions Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 8 / 35

Intra-Picture Prediction / Spatial Intra Prediction Spatial Intra Prediction in H.264 | MPEG-4 AVC (for 4 × 4 Blocks) vertical prediction horizontal prediction DC prediction a �� � = a = 8 average − 90 ◦ 0 ◦ a b c diagonal down-left prediction diagonal down-right prediction vertical-right prediction � � = a + 2 b + c + 2 ≫ 2 a b − 135 ◦ − 45 ◦ − 63 . 4 ◦ � � = a + 3 b + 2 ≫ 2 horizontal-down prediction vertical-left prediction horizontal-up prediction a b = � a + b + 1 � ≫ 1 − 27 . 6 ◦ − 117 . 6 ◦ 27 . 6 ◦ similar 9 modes for 8 × 8 blocks (with additional border smoothing); 4 modes for 16 × 16 blocks (DC, hor., ver., plane) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 9 / 35

Intra-Picture Prediction / Spatial Intra Prediction Spatial Intra Prediction in H.265 | MPEG-H HEVC -32 0 32 -32 DC prediction mode : − 135 ◦ � � � 1 � � = N + 2 N Intra Prediction in H.265 | MPEG-H HEVC 0 Unique intra prediction for all block sizes (4 × 4 to 32 × 32) 35 spatial intra prediction modes 33 directional prediction modes (slopes of k / 32) DC prediction mode (similar to H.264 | AVC) Planar prediction mode (smooth prediction signal) 45 ◦ 32 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 10 / 35

Intra-Picture Prediction / Spatial Intra Prediction Angular Intra Prediction in H.265 | MPEG-H HEVC Mainly vertical modes (modes 18 – 34) example : 1 Fill not existing border samples mode 21 Replace with closest existing border sample 2 Filtering of border samples Typically: Use (1,2,1) filter (as in H.264) No filtering for close to vertical modes 3 Create virtual top border Copy from left border in prediction direction 4 Predict samples from virtual top border Predict from virtual top border in prediction direction (using linear interpolation) Mainly horizontal modes (modes 2 – 17) Same signal processing (for transposed data) linear interpolation No filtering for close to horizontal modes between neighboring Use virtual left border border samples Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 11 / 35

Intra-Picture Prediction / Spatial Intra Prediction Planar Prediction Mode in H.265 | MPEG-H HEVC Basic Idea � L ′ y , T ′ x , R ∗ y , B ∗ � s x , y = bilinear ˆ x Not all blocks fit an directional model R ∗ y = linear � T ′ N − 1 , C ∗ � DC mode: Coarse approximation (order 0) B ∗ L ′ N − 1 , C ∗ � y = linear � Planar mode : Smooth approximation Bilinear interpolation of sample values C ∗ = 1 � T ′ N − 1 + L ′ � N − 1 Linear interpolated right-bottom border 2 Planar Mode in HEVC Simplified design without border interpolation ˆ copy h [ x , y ] = ( N − 1 − x ) · s ′ [ − 1 , y ] + ( 1 + x ) · s ′ [ N − 1 , − 1 ] v [ x , y ] = ( N − 1 − y ) · s ′ [ x , − 1 ] + ( 1 + y ) · s ′ [ − 1 , N − 1 ] ˆ 1 � � ˆ s [ x , y ] = ˆ h [ x , y ] + ˆ v [ x , y ] copy 2 N Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 12 / 35

Intra-Picture Prediction / Spatial Intra Prediction Illustration of Intra Prediction Modes in H.265 | MPEG-H HEVC 26 · · · · · · 18 34 ? . . . 10 . . . 2 35 intra prediction modes : 0: planar mode 1: DC mode 2: angular mode ( + 45 ◦ ) · · · 10: angular mode (0 ◦ ): horizontal · · · 18: angular mode ( − 45 ◦ ) · · · 26: angular mode ( − 90 ◦ ): vertical · · · 34: angular mode ( − 135 ◦ ) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Intra Prediction & Picture Partitioning 13 / 35