In the name of Allah the compassionate, the merciful
Digital Video Systems S. Kasaei S. Kasaei Room: CE 307 Department of Computer Engineering Sharif University of Technology E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei Lab. Website: http://mehr.sharif.edu/~ipl
Acknowledgment Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1 st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001].
Chapters 9 & 11 Video Coding using Motion Compensation
Outline Block-Based Hybrid Video Coding � Overview of Block - B a sed Hybrid Video Coding � Overlapped Block Motion Compensation � Coding Mode Selection & Rate Control � Loop Filtering � Scalable Video Coding � Motivation for Scalable Coding � Basic Modes of Scalability � Scalability in MPEG - 2 � Fine Granularity Scalability in MPEG - 4 � Kasaei 6
Characteristics of Typical Videos Frame t-1 Frame t Adjacent frames are similar & changes are due to object or camera motion. Kasaei 7
Key Ideas in Video Compression Predicts a new frame from a previous frame & only � codes the prediction error. Prediction error cab be coded using the DCT method. � Prediction errors have smaller energy than the � original pixel values & can be coded with fewer bits. Those regions that cannot be predicted well will be � coded directly using DCT. Works on each macroblock (MB) ( 16x16 pixels) � independently to reduce the complexity. Motion compensation is done at the MB level. � DCT coding of error is done at block level ( 8x8 pixels). � Kasaei 8
Different Coding Modes Kasaei 9
Temporal Prediction No Motion Compensation: � Works well in stationary regions. � $ ( , f t m n , ) f t ( − 1 , m n , ) = Uni - d irectional Motion Compensation: � Does not work well for uncovered regions by object motion. � $ ( , f t m n , ) f t ( 1 , m d , n d ) = − − − x y Bi - d irectional Motion Compensation: � Can better handle covered/uncovered regions. � $ ( , f t m n , ) w f t ( 1 , m d , n d ) = − − − b b x , b y , w f t ( 1 , m d , n d ) + + − − f f x , f y , Kasaei 10
Temporal Prediction � Although bi-directional prediction can improve prediction accuracy & consequently coding efficiency, it incurs encoding delay & is typically not used in real-time applications. � H.261/H.263 use only uni-directional prediction & a restricted bi-directional prediction (PB-mode). � MPEG employs both uni- & bi-directional prediction. Kasaei 11
Encoder Block Diagram of a Block-Based Hybrid Video Coder [Hybrid: a combination of motion-compensated temporal prediction + transform coding.] Kasaei 12
Decoder Block Diagram Kasaei 13
Block-Based Hybrid Video Coding The encoder must emulate the decoder � operation to deduce the same reconstructed frame as the decoder. Frame types: � When a frame is coded entirely in the intra-mode � � I-frame. When a previous frame is used for prediction in the � inter-mode � P-frame. When a weighted sum of previous & following frame � is used for prediction in the inter-mode � B-frame. The mode information, MVs, & other side � information (picture format, block location, …) are coded using VLC. Kasaei 14
Block-Based Video Partitions Kasaei 15
MB Structure in 4:2:0 Color Format 4 of 8x8 Y blocks. 1 of 8x8 Cb blocks. 1 of 8x8 Cr blocks. Kasaei 16
Block Matching Algorithm for Motion Estimation MV Search Region Frame t-1 Frame t (reference frame) (predicted frame) Kasaei 17
Macroblock Coding in I-Mode DCT transform each 8x8 DCT block. Quantize DCT coefficients (with properly chosen quantization matrices). Zig - z ag order & run - l ength code quantized DCT coefficients. Kasaei 18
Macroblock Coding in P-Mode Estimate one MV for each macroblock ( 16x16 ). Depending on the motion compensation error, determine the coding mode (intra, inter - w ith - n o - MC, inter - with - MC, etc.) Original values (for intra - mode) or motion compensation errors (for inter - mode) in each of the DCT blocks ( 8x8 ) are DCT transformed, quantized, zig - z ag/alternate scanned, & run- length coded. Kasaei 19
Macroblock Coding in B-Mode Same as for the P - m ode, except a macroblock can � be predicted from a previous frame, a following one, or both. v f v b Kasaei 20
Overlapped Block Motion Compensation (OBMC) Conventional block motion compensation: � One best matching block is found from a reference � frame. The current block is replaced by the best matching � block. OBMC: � Each pixel in the current block is predicted by a � weighted average of several corresponding pixels in the reference frame. The corresponding pixels are determined by the � MVs of the current as well as adjacent MBs. The weights for each corresponding pixel depends � Kasaei 21 on the expected accuracy of the associated MV.
OBMC using 4-Neighboring MBs weight assigned to estimated value based on MV ( d m,k ). should be inversely proportional to the distance between x & the center of ([h m,1 & h m,4 ] > [h m,2 & h m,3 ]) Kasaei 22
Optimal Weighting Design Convert to an optimization problem: � Minimize: � Subject to: � Optimal weighting functions: � autocorrelation Kasaei 23 cross-correlation
How to Determine MVs with OBMC Option 1: using conventional BMA, minimize the � prediction error (MAD) within each MB independently. Option 2: minimize the prediction error � assuming OBMC: � Solve the MV for the current MB while keeping the MVs for the neighboring MBs found in the previous iterations. Kasaei 24
How to Determine MVs with OBMC Option 3: Using a weighted error criterion over � a larger block: window function Kasaei 25
Weighting Coefficients used in H.263 Kasaei 26
Window Function Corresponding to H.263 Weights for OBMC Kasaei 27
Coding Parameter Selection Coding modes: � Intra vs. inter, quantization parameter (QP) for � each MB, ME method ( � different rates). Rate-distortion optimized selection, given a � target rate: Minimize the distortion, subject to the target rate � constraint: simplified version: The optimal mode is such that each MB works at the same R-D slope:
Rate Control Rate control: � How to code a video so that the resulting bit � stream satisfies a target bit rate? For pleasant visual perception, video should have � a constant quality. But, the coding method necessarily yields variable � bit rate. So, at least the bit rate should be constant when � averaged over a short period. Rate control is also necessary when the video is to � be sent over a constant bit rate (CBR) channel. The fluctuation within the period can be smoothed � by a buffer at the encoder output. Kasaei 29
Rate Control � Rate control accomplished steps: Step 1) Determine the target average bit rate at � the frame, GOB, & MB level, based on the current buffer fullness. Step 2) Satisfy frame level target rate by varying � frame rate (skip frames when necessary). Step 3) Satisfy GOB/MB level target rate by � varying the coding mode & QP at each MB. (= Rate-distortion optimized mode selection.) Kasaei 30
Loop Filtering Errors in previously reconstructed frames � accumulate over time with motion compensated temporal prediction. Error propagation leads to: � Reduction of prediction accuracy. � Increase of the bitrate needed for coding new � frames. Kasaei 31
Loop Filtering Loop filtering: � Filters the reference frame before using it for � prediction. Can be embedded in the motion compensation � loop. Half-pel motion compensation. � OBMC. � Loop filtering can significantly improve coding � efficiency. For theoretically optimal design of loop filters � see text. Kasaei 32
Scalable Coding (Ch. 11) Scalability refers to the capability of recovering � physically meaningful image (or video) information by decoding only partial compressed bit stream. It is used when users try to access the same video � through different communication links (bandwidth scalability). A scalable stream can also offer adaptivity to � varying channel error characteristics & computing power at the receiving terminal. This includes: quality, spatial, temporal, & � frequency scalability. Kasaei 33
Scalable Coding Motivation: � Real networks are heterogeneous in rate. � Streaming video from home ( 56 kbps) using � modem vs. corporate LAN ( 10-100 mbps). Scalable video coding: � Ideal goal (embedded stream): creating a bit � stream that can be accessed at any rate. Practical video coder: � layered coder: base-layer provides basic quality, � successive layers refine the quality incrementally. Coarse granularity: (typically known as layered � coder). Kasaei 34 Fine granularity (FGS). �
Combined Spatial/Quality Scalability Kasaei 35
Recommend
More recommend
