In the name of Allah the compassionate, the merciful the compassionate, the merciful
Digital Video Processing S. Kasaei S S S. Kasaei asae asae Room: CE 307 Department of Computer Engineering Sharif University of Technology Sh if U i it f T h l E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei p g p Lab. Website: http://ipl.ce.sharif.edu
Acknowledgment Most of the slides used in this course have been provided by: Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1 st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001].
Chapters 9 & 11 Video Coding Video Coding using M ti Motion Compensation C ti
Outline Block-Based Hybrid Video Coding � Overview of Block-Based Hybrid Video Coding � Overlapped Block Motion Compensation � Coding Mode Selection & Rate Control � L Loop Filtering Filt i � Scalable Video Coding � Motivation for Scalable Coding Motivation for Scalable Coding � � Basic Modes of Scalability � Scalability in MPEG-2 Scalability in MPEG 2 � � Fine Granularity Scalability in MPEG-4 � Kasaei 6
Characteristics of Typical Videos f Frame t-1 Frame t Adjacent frames are similar & changes are due to object or camera motion (on a static background). Kasaei 7
Key Ideas in Video Compression C Predicts a new frame from a previous frame & only � codes the prediction error. P Prediction error cab be coded using the DCT method. di ti b b d d i th DCT th d � Prediction errors have smaller energy than the � original pixel values & can be coded with fewer bits. original pixel values & can be coded with fewer bits. Those regions that cannot be predicted well, will be � coded directly using DCT. Works on each macroblock (MB) ( 16x16 pixels) � independently to reduce the complexity. Motion compensation is done at MB level Motion compensation is done at MB level. � � DCT coding of error is done at block level ( 8x8 pixels). � Kasaei 8
Different Coding Modes ff C Kasaei 9
T Temporal Prediction l P di ti No Motion Compensation: No Motion Compensation: � � Works well in stationary regions. � $ ( , = − 1 f f t m n ( , , ) , ) f f t ( ( , , m n , ) , ) Uni-directional Motion Compensation: � Does not work well for uncovered regions by object motion. � $ ( , = − − − f t m n , ) f t ( 1 , m d , n d ) x y Bi-directional Motion Compensation: p � Can better handle covered/uncovered regions. � $ ( , = − − − f t m n , ) w f t ( 1 , m d , n d ) b b x , b y , + + − − w f t ( 1 , m d , n d ) f f x , f y , Kasaei 10
T Temporal Prediction l P di ti Kasaei 11
Temporal Prediction � Although bi-directional prediction can improve prediction accuracy & consequently coding efficiency, it incurs encoding delay & is typically not used in real-time applications. � H.261/H.263 use only uni-directional H 261/H 263 l i di ti l prediction & a restricted bi-directional prediction (PB mode) prediction (PB-mode). � MPEG employs both uni- & bi-directional prediction prediction. Kasaei 12
Encoder Block Diagram of a Block-Based Hybrid Video Coder f Bl k B d H b id Vid C d [Hybrid: a combination of motion-compensated temporal prediction + transform coding.] Kasaei 13
Decoder Block Diagram Kasaei 14
Block-Based Hybrid Video Coding C The encoder must emulate the decoder � operation to deduce the same reconstructed frame as the decoder. Frame types: � When a frame is coded entirely in the intra-mode � � I-frame. a e When a previous frame is used for prediction in the � inter-mode � P-frame. When a weighted sum of previous & following frame g p g � is used for prediction in the inter-mode � B-frame. The mode information, MVs, & other side � information (picture format, block location, …) (p ) are coded using VLC. Kasaei 15
Bl Block-Based Video Partitions k B d Vid P i i Kasaei 16
MB Structure i in 4:2:0 Color Format 4 2 0 C l F t 4 of 8x8 Y blocks. 1 of 8x8 Cb blocks. 1 of 8x8 Cr blocks. Kasaei 17
Block Matching Algorithm for Motion Estimation for Motion Estimation MV Search Region Frame t 1 Frame t-1 Frame t Frame t (reference frame) (predicted frame) Kasaei 18
Macroblock Coding i in I-Mode I M d DCT transform each 8x8 DCT block. Quantize DCT coefficients (with properly chosen quantization matrices) (with properly chosen quantization matrices). Zig-zag order & run-length code quantized DCT coefficients. Kasaei 19
Macroblock Coding i in P-Mode P M d Estimate one MV for each macroblock ( 16x16 ). Depending on the motion compensation error, determine the coding mode (intra, inter-with-no-MC, inter- with MC etc ) with-MC, etc.) Original values (for intra-mode) or motion compensation g ( ) p errors (for inter-mode) in each of the DCT blocks ( 8x8 ) are DCT transformed, quantized, zig-zag/alternate scanned & run-length coded scanned, & run length coded. Kasaei 20
Macroblock Coding i in B-Mode B M d Same as for the P-mode, except that a macroblock � can be predicted from a previous frame, a following one or both one, or both. v f v f v b Kasaei 21
Overlapped Block Motion C Compensation (OBMC) (O C) Conventional block motion compensation: Conventional block motion compensation: � � One best matching block is found from a reference � frame. The current block is replaced by the best matching � block. OBMC: OBMC: � � Each pixel in the current block is predicted by a � weighted average of several corresponding pixels in the reference frame. The corresponding pixels are determined by the � MVs of the current as well as adjacent MBs MVs of the current as well as adjacent MBs. The weights for each corresponding pixel depends � Kasaei 22 on the expected accuracy of the associated MV.
OBMC using 4-Neighboring MBs weight assigned to estimated value based on MV ( d m,k ). should be inversely should be inversely proportional to the distance between x & the center of ([h m,1 & h m,4 ] > [h m,2 & h m,3 ]) ( ) K: total no. of neighboring blocks Bm: block Kasaei 23
Optimal Weighting Design Convert to an optimization problem: � Minimize: � Subject to: Subject to: � � Optimal weighting functions: � autocorrelation autocorrelation Kasaei 24 cross-correlation
How to Determine MVs with OBMC O C Option 1: using conventional BMA, minimize the � prediction error (MAD) within each MB i d independently. d tl Option 2: minimize the prediction error � assuming OBMC: assuming OBMC: � Solve the MV for the current MB while keeping the MVs for the neighboring MBs p g g g found in the previous iterations. Kasaei 25
How to Determine MVs with OBMC O C Option 3: Using a weighted error criterion over � a larger block: window function function Kasaei 26
Weighting Coefficients used in H.263 Top or Left or Current Bottom Right Block Kasaei 27
Window Function Corresponding t to H.263 Weights for OBMC H 263 W i ht f OBMC Kasaei 28
C Coding Parameter Selection S Coding modes Coding modes: � Intra vs. inter, quantization parameter (QP) for � each MB, ME method ( � different rates). each MB, ME method ( � different rates). Rate-distortion optimized selection, given a � target rate: Minimize the distortion, subject to the target rate � constraint: simplified version: The optimal mode is such that each MB works at the same R-D slope:
Rate Control C Rate control: � How to code a video so that the resulting bit � stream satisfies a target bit rate? g For pleasant visual perception, video should have � a constant quality. But, the coding method necessarily yields variable g y y � bit rate. So, at least the bit rate should be constant when � averaged over a short period. Rate control is also necessary when the video is to � be sent over a constant bit rate (CBR) channel. The fluctuation within the period can be smoothed � b by a buffer at the encoder output. b ff t th d t t Kasaei 30
Rate Control C � Rate control accomplished steps: Step 1) Determine the target average bit rate at � the frame GOB & MB level based on the the frame, GOB, & MB level, based on the current buffer fullness. Step 2) Satisfy frame level target rate by varying � frame rate (skip frames when necessary). f ( ki f h ) Step 3) Satisfy GOB/MB level target rate by � varying the coding mode & QP at each MB. y g g (= Rate-distortion optimized mode selection.) Kasaei 31
Loop Filtering Errors in previously reconstructed frames � accumulate over time with motion compensated t temporal prediction. l di ti Error propagation leads to: � Reduction of prediction accuracy Reduction of prediction accuracy. � Increase of the bitrate needed for coding new � frames. Kasaei 32
Recommend
More recommend
