Motion Estimation for Video Coding Motion-Compensated Prediction - PowerPoint PPT Presentation

Motion Estimation for Video Coding  Motion-Compensated Prediction  Bit Allocation  Motion Models  Motion Estimation  Efficiency of Motion Compensation Techniques T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 1

Hybrid Video Encoder Coder Control Control Data [ , , ] s x y t [ , , ] u x y t Intra-Frame DCT DCT Coder - Coefficients Decoder Intra-Frame Decoder ' [ , , ] u x y t ' [ , , ] s x y t 0 Motion- Compensated Intra/Inter Predictor ˆ [ , , ] s x y t Motion Data Motion Estimator T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 2

Motion-Compensated Prediction Previous frame Stationary background Moving  t object Current frame x y time t   d   Displaced x   d object   y Prediction for the luminance signal s [ x,y,t ] within the moving object:       ˆ [ , , ] ( , , ) s x y t s x d y d t t x y T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 3

Motion-Compensated Prediction: Example Partition of frame 2 into blocks Frame 1 s [ x,y,t-1 ] (previous) Frame 2 s [ x,y,t ] (current) (schematic) Size of Blocks Accuracy of Motion Vectors Frame 2 with Difference between motion- Referenced blocks in frame 1 displacement vectors compensated prediction and current frame u [ x,y,t ] T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 4

Motion Models  Motion in 3-D space corresponds to displacements in the image plane  Motion compensation in the image plane is conducted to provide a prediction signal for efficient video compression  Efficient motion-compensated prediction often uses side information to transmit the displacements  Displacements must be efficiently represented for video compression  Motion models relate 3-D motion to displacements assuming reasonable restrictions of the motion and objects in the 3-D world         Motion Model ( , , ), ( , , ) d x x f a x y d y y f b x y x x y y , x y : location in previous image   , : location in current image x y , a b : vector of motion coefficients , : displacements d d x y T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 5

Representation of Video Signal Decoded video signal is given as     ˆ [ , , ] [ , , ] [ , , ] s x y t s x y t u x y t Motion-compensated Prediction residual signal prediction signal  1 M         ˆ  j   [ , , ] ( , , ) s x y t s x d y d t t ( , ) ( , ) u x y c x y x y j  0 j   1 1 N N         ( , ), ( , ) d a x y d b x y x i i y i i   0 0 i i Transmitted residual parameters Transmitted motion parameters   ( ), ( ,...) c c R u f c    0 ( , ), ( ,...), ( ,...) R m f a b a a b b 0 0 T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 6

Rate-Constrained Motion Estimation Motion Bit-rate vector rate D R m Prediction error rate R u Displacement error variance dD dD  Optimum trade-off:    , R R R m u dR dR m u  Displacement error variance can be influenced via • Block-size, quantization of motion parameters • Choice of motion model T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 7

Lagrangian Optimization in Video Coding  A number of interactions are often neglected • Temporal dependency due to DPCM loop • Spatial dependency of coding decisions • Conditional entropy coding  Rate-Constrained Motion Estimation [Sullivan, Baker 1991]:   min D R m m m Distortion after Number of bits Lagrange motion compensation for motion vector parameter  Rate-Constrained Mode Decision [Wiegand, et al. 1996]:   min D R Distortion after Number of bits Lagrange reconstruction for coding mode parameter T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 8

Motion Models   1 1 N N         ( , ), ( , ) d a x y d b x y x i i y i i   0 0 i i    Translational motion model 0 , d a d b 0 x y  4-Parameter motion model: translation, zoom (isotropic Scaling),    rotation in image plane d a a x a y 0 1 2 x    d b a x a y 0 2 1 y    d a a x a y  Affine motion model: 0 1 2 x    d b b x b y 0 1 2 y       2 2  Parabolic motion model d a a x a y a x a y a xy 0 1 3 2 6 5 x       2 2 d b b x b y b x b y b xy 0 1 3 2 6 5 x T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 9

Impact of the Affine Parameters 150   100 d x a translation 50 0 y 0 -150 -100 -50 0 50 100 150 -50 -100 -150 x 150 150 100 100   d x a x scaling 50 50 1 y 0 y 0 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -50 -50 -100 -100 -150 -150 x x 150   100 d x a y sheering 50 3 y 0 -150 -100 -50 0 50 100 150 -50 -100 -150 x T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 10

Impact of the Parabolic Parameters 150 d x  100 2 a 2 x 50 y 0 -150 -100 -50 0 50 100 150 -50 -100 -150 x 150 150 d x  100 100 2 a 6 y 50 50 y 0 y 0 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -50 -50 -100 -100 -150 -150 x x 150 100  d x a xy 50 5 y 0 -150 -100 -50 0 50 100 150 -50 -100 -150 x T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 11

Differential Motion Estimation  Assume small displacements d x ,d y :   ˆ ( , , ) ( , , ) ( , , , , ) u x y t s x y t s x y t d d x y         ( , , ) ( , , ) s x y t t s x y t t          ( , , ) ( , , ) s x y t s x y t t d d   x y x y Horizontal and vertical Displace frame difference gradient of image signal S  Aperture problem: several observations required  Inaccurate for displacements > 0.5 pel  multigrid methods, iteration  Minimize By Bx  2 min ( , , ) u x y t   1 1 y x T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 12

Gradient-Based Affine Refinement  Displacement vector field is represented as     x a a x a y 1 2 3  Combination     y b b x b y 1 2 3     s s            ( , , ) ( , , ) ( , , ) ( ) ( ) u x y t s x y t s x y t t a a x a y b b x b y  1 2 3  1 2 3 x y   a   1 yields a system of linear equations:   a 2                 s s s s s s a         3 , , , , , , u s s x y x y         b     x x x y y y 1   b   2     b 3  System can be solved using, e.g., pseudo-inverse, By by minimizing Bx  2 arg min ( , , ) u x y t , , , , ,   a a a b b b 1 1 y x 1 2 3 1 2 3 T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 13

Block-matching Algorithm • Subdivide current frame search range in into blocks. previous frame S k  1 • Find one displacement vector for each block. • Within a search range, find a “best match” that minimizes an error measure. • Intelligent search strategies can reduce computation. block of current S k frame T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 14

Block-matching Algorithm Current Frame Previous Frame Measurement window is Block of pixels is selected compared with a shifted block as a measurement window of pixels in the other image, to determine the best match T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 15

Block-matching Algorithm Current Frame Previous Frame . . . process repeated for another block. T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 16

Error Measures for Block-matching  Mean squared error (sum of squared errors) By Bx         2 ( , ) [ ( , , ) ( , , )] SSD d d s x y t s x d y d t t x y x y   1 1 y x  Sum of absolute differences By Bx         ( , ) | ( , , ) ( , , ) | SAD d d s x y t s x d y d t t x y x y   1 1 y x  Approximately same performance  SAD less complex for some architectures T. Wiegand / B. Girod: EE398A Image and Video Compression Motion estimation no. 17