Image and Video Coding: Encoder Control D D = - R d R Problem - PowerPoint PPT Presentation

Image and Video Coding: Encoder Control D D = - λ R d R

Problem Statement / Scope of Image and Video Coding Standards Image and Video Coding input output pre- bitstream post- image/video image/video processing processing encoder decoder samples samples scope of standards Interoperability and Standards Bitstream generated by an encoder should be reliably decodable by decoders of other manufacturers Image and video coding standards define “interface” between encoder and decoder Scope of Image and Video Coding Standards 1 Bitstream syntax (including constraints for transmitted and derived parameters) 2 Decoding process (example decoding process for conforming bitstreams) No guarantee of image/video quality Coding efficiency is determined by all components Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 2 / 35

Problem Statement / Coding Efficiency Coding Efficiency: Trade-Off between Bit Rate and Reconstruction Quality Bit Rate Use average bit rate in our comparisons (count bits in bitstream) number of bits in bitstream for the video sequence � � R = in bits/s nominal duration of the video sequence (in seconds) Reconstruction Quality Human perception of visual quality: Difficult to evaluate (subjective tests) Use mean squared error (MSE) and peak signal-to-noise ratio (PSNR) 1 � 2 � � s ′ [ x , y ] − s [ x , y ] MSE = W · H ∀ x , y � s 2 � maximum sample value: s max = 2 B − 1 � � PSNR [dB] = 10 · log 10 max MSE Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 3 / 35

Problem Statement / Coding Efficiency Coding Efficiency: Rate-Distortion Curves & Bit-Rate Savings 70 41 codec B 60 bit rate saving of R B codec B vs. codec A 40 bit-rate saving [%] codec A 50 PSNR [dB] 39 40 R A 38 30 example target quality example 37 20 B BA = (R A - R B ) / R A ( 39 dB ) target quality ( 39 dB ) 36 10 35 0 0 1000 2000 3000 4000 5000 36 37 38 39 40 41 bit rate [kbit/s] PSNR [dB] Rate-Distortion Curve Measure average bit rate and average PSNR for multiple operation points Bit-Rate Savings Determine bit-rate savings using interpolated rate-distortion curves � � B BA = ( R A − R B ) / R A codec B relative to codec A Average bit-rate savings : Can be calculated by integrating curve Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 4 / 35

Problem Statement / Impact of Encoder Control Coding Efficiency of Image and Video Codecs original bitstream reconstructed image/video image/video encoder decoder samples samples syntax & decoding process Bitstream Syntax and Decoding Process Specify syntax features and supported coding tools Determine maximum achievable coding efficiency Image/Video Encoder Lot of freedom to choose coding parameters (modes, motion vectors, quantization indexes, ...) Encoding process determines coding efficiency of a bitstream for given syntax and decoder Main Encoding Problem Select coding parameters such that the coding efficiency is maximized Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 5 / 35

Problem Statement / Impact of Encoder Control Example: Which Quantization Method is Better ? abs. quant. indexes | q k | abs. transform coeffs. | u k | abs. quant. indexes | q k | 9 5 2 0 0 0 0 0 9 5 2 0 0 0 0 0 93 46 24 4 2 1 3 4 simple rounding : alternative 4 0 0 0 0 0 0 1 4 0 0 0 0 0 0 0 37 4 4 3 1 0 2 6 quantization � u k � q k = round 1 0 0 0 0 0 0 0 14 4 2 3 2 3 3 1 1 0 0 0 0 0 0 0 method ∆ 1 0 0 0 0 0 0 0 7 3 1 3 1 2 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 1 2 1 4 0 3 2 0 0 0 0 0 0 0 0 quantization step size quantization step size 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 2 1 0 0 2 1 1 0 0 0 0 0 0 0 0 ∆ = 10 3 1 1 1 2 0 2 1 ∆ = 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 2 0 1 3 1 0 0 0 0 0 0 0 0 0 run-level pairs (absolute levels): excerpt of MPEG-2 table run-level pairs (absolute levels): (0,9) (0,5) (0,4) (0,1) (1,2) (3,1) (32,1) (eob) (run, level) codeword (s = sign) (0,9) (0,5) (0,4) (0,1) (1,2) (3,1) (eob) (eob) 10 distortion (SSD): ( 0 , ± 1 ) 11s distortion (SSD): ( 0 , ± 4 ) 0000 110s ( u k − ∆ · q k ) 2 = 371 ( u k − ∆ · q k ) 2 = 391 � � D 1 = D 2 = (- 0.23 dB) ( 0 , ± 5 ) 0010 0110 s ∀ k ( 0 , ± 9 ) 0000 0001 1101 s ∀ k ( 1 , ± 2 ) 0001 10s number of bits : number of bits : ( 3 , ± 1 ) 0011 1s R 1 = 13 + 9 + 8 + 3 + 7 + 6 + 24 + 2 (escape) 0000 01 (+18 bits) R 1 = 13 + 9 + 8 + 3 + 7 + 6 + 2 R 1 = 72 bits note : there is no ( 32 , ± 1 ) pair R 2 = 48 bits (- 33.3 %) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 6 / 35

Problem Statement / Impact of Encoder Control Quantization Example: Impact on Coding Efficiency PSNR alternative simple quantization quantization method same quantization step size ∆ bit rate Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 7 / 35

Problem Statement / Impact of Encoder Control Example: Motion Estimation and Mode Decision reconstructed reference picture Motion Estimation Goal: Find matching block in reference picture Match with minimum distortion : SSD distortion: D = 273 motion vector: m = ( 30 , 38 ) 22 bits (EG0) Alternative match : SSD distortion: D = 295 ( − 0 . 34 dB ) motion vector: m = ( 2 , − 1 ) 8 bits (-64 %) Mode Decision How to decide between intra and inter coding for a block? How to decide between different coding modes? current picture Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 8 / 35

Lagrangian Encoder Control / Encoding Problem General Encoding Problem in Image and Video Coding Given Bitstream syntax (format for transmitting coding parameters) Decoding process (algorithm for reconstructing pictures) Encoding Problem Choose coding parameters (and thus the bistream) in a way that coding efficiency is maximized Selection of coding modes Selection of motion parameters Selection of quantization indexes Criterion for Coding Efficiency Need to consider both Quality (or distortion) of reconstructed pictures Bit rate of resulting bitstream Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 9 / 35

Lagrangian Encoder Control / Encoding Problem Encoding Problem: Mathematically Formulation Formulation of Encoding Problem Find conforming bitstream b ∗ that minimizes the distortion D for the given input image/video s v and has a bit rate R that does not exceed a bit rate budget R B b ∗ = arg min � � s v , s ′ v ( b ) subject to R ( b ) ≤ R B D ∀ b ∈B with s v : samples of entire image/video s ′ v ( b ) : reconstructed image/video for bitstream b D ( s v , s ′ v ) : distortion between original and reconstructed image/video B : set of all conforming bitstreams R B : maximum available bit rate for encoding task Impossible to find optimal solution (extremely large parameter space) Split into smaller sub-problems Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 10 / 35

Lagrangian Encoder Control / Lagrangian Optimization Lagrangian Optimization for Discrete Sets Constrained Optimization Problem Consider set of samples s (block, picture) and discrete vector of coding parameters p � � Constrained problem for given rate budget R B , with D ( p ) = D s , s ′ ( p ) p opt ( R B ) = arg min D ( p ) subject to R ( p ) ≤ R B p Varying R B : Optimal coding parameter vectors { p opt } Unconstrained Optimization Problem Using Lagrange multipliers λ ≥ 0, we obtain the unconstrained problem p ∗ λ = arg min D ( p ) + λ · R ( p ) p Cannot find all optimal solutions { p opt } But: Each solution p ∗ λ is an optimal solution , { p ∗ λ } ⊆ { p opt } Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 11 / 35

Lagrangian Encoder Control / Lagrangian Optimization Optimality of Lagrangian Approach Consider solution p ∗ λ for a particular value of λ , with λ ≥ 0 By definition, we have D ( p ) + λ · R ( p ) ≥ D ( p ∗ λ ) + λ · R ( p ∗ ∀ p , λ ) D ( p ) − D ( p ∗ R ( p ∗ � � λ ) ≥ λ · λ ) − R ( p ) Since λ ≥ 0, the above inequality implies R ( p ) ≤ R ( p ∗ D ( p ) ≥ D ( p ∗ ∀ p : λ ) , λ ) Hence, p ∗ λ is a solution of the constrained problem p ∗ R ( p ) ≤ R B = R ( p ∗ λ = arg min D ( p ) subject to λ ) p Each solution of the unconstrained optimization problem is also a solution of original constrained optimization problem Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 12 / 35

Lagrangian Encoder Control / Lagrangian Optimization Illustration of Lagrangian Optimization D convex hull D+ λ R tangent with convex hull slope - λ solution of constrained D = - λ R d problem d (1+ λ 2 ) 0.5 solution of solution of tangent parallel unconstrained unconstrained to R-axis problem problem R B R R Solutions of Lagrangian optimization problem minimize distance d to lines D = − λ · R Subset { p ∗ λ } lies on convex hull of area of all possible rate-distortion points Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Encoder Control 13 / 35