In the name of Allah In the name of Allah In the name of Allah In - PowerPoint PPT Presentation

Basic Macroblock Basic Macroblock Coding Structure Coding Structure 28 Coder Coder Input Video Control Control Signal Data Transform/ Quant. Quant. Scal./Quant. S l /Q t - Transf. coeffs Decoder Scaling & Inv. Split into Transform Macroblocks (16x16 pixels) (16x16 pixels) Entropy Entropy Coding De-blocking Filter Intra-frame Prediction Output Motion- Video Compensation Signal Intra/Inter Motion Data Motion Kasaei Estimation

Encoder (Forward Path) Encoder (Forward Path) 29 � An input frame or field F n is processed in units of MBs. � Each MB is encoded in intra or inter mode. E h MB i d d i i i d � For each block in the MB, a prediction PRED (marked ‘ P ’ in the figure) is formed based on reconstructed picture samples. Kasaei

Encoder (Forward Path) Encoder (Forward Path) 30 � In Intra mode: � PRED is formed from samples in the current slice that have been previously encoded, decoded and reconstructed. i l d d d d d d d � uF’ n in the figures; note that unfiltered samples are used to form PRED. � In Inter mode: � PRED is formed by motion-compensated prediction from one or y p p more reference picture(s) selected from a set of reference pictures. Kasaei

Encoder (Forward Path) Encoder (Forward Path) 31 P is subtracted from the current block to produce D n (a residual block) that is � transformed and quantized to give X. X is a set of quantized transform coefficients which are reordered and entropy X i t f ti d t f ffi i t hi h d d d t � encoded. Entropy-encoded coefficients, together with side information (prediction modes, � quantization parameter, motion vector information, etc.) form the compressed bitstream. Compressed bitstream is passed to an NAL for transmission or storage. � Kasaei

Encoder (Reconstruction Path) Encoder (Reconstruction Path) 32 Encoder decodes (reconstructs) every MB to provide a reference for further � predictions. Coefficients X are scaled ( Q − 1 ) and inverse transformed ( T − 1 ) to produce a � difference block D’ n . Prediction block P is added to D’ n to create a reconstructed block uF’ n � A decoded version of the original block. � u indicates that it is unfiltered. u indicates that it is unfiltered. � � A filter is applied to reduce the effects of blocking distortion and the � reconstructed reference picture is created from a series of blocks F’ n . Kasaei

Decoder Decoder 33 Decoder receives a compressed bitstream from the NAL. � Entropy decodes the data elements to produce a set of quantized coefficients X . Entropy decodes the data elements to produce a set of quantized coefficients X . � These are scaled and inverse transformed to give D’ n � I dentical to the D’ n shown in the Encoder. � Using the header information decoded from the bitstream, the decoder creates a � prediction block PRED. Identical to the original prediction PRED formed in the encoder Identical to the original prediction PRED formed in the encoder. � � PRED is added to D’ n to produce uF’ n , which is filtered to create each decoded � block F’ n . Kasaei

Intra Intra- -Prediction Prediction 34 � Motivation: Intra-frames are natural images, so they exhibit strong spatial correlation. hibi i l l i � Macro-blocks in intra-coded frames are predicted based on Macro blocks in intra coded frames are predicted based on previously-coded macro-blocks. � Above and (or to the left of) the current block. � Macro-block may be divided into 16 4x4 sub-blocks which are � Macro-block may be divided into 16, 4x4 sub-blocks which are predicted in a cascading fashion. � 9 modes for 4x4 and 4 modes for 16x16 size � 9 modes for 4x4 and 4 modes for 16x16 size. Kasaei

Intra- Intra -Prediction Prediction 35 Coder Coder Input Video Control Control Signal Data Transform/ Quant. Quant. Scal./Quant. S l /Q t - Transf. coeffs Decoder Scaling & Inv. Split into Transform Macroblocks (16x16 pixels) (16x16 pixels) Entropy Entropy Coding De-blocking Filter Intra-frame Prediction Output Motion- Video Compensation Signal Intra/Inter Motion Data Motion Kasaei Estimation

Intra- Intra -Prediction Prediction 36 Coder Coder � Directional spatial prediction Input Video Control (9 types for luma, 4 chroma) Control Signal Data Q A B C D E F G H Transform/ Quant. Quant. I a b c d I a b c d S Scal./Quant. l /Q t - Transf. coeffs J e f g h Decoder Scaling & Inv. K i j k l Split into Transform L m n o p Macroblocks M (16x16 pixels) (16x16 pixels) Entropy Entropy N Coding 0 O De-blocking 7 P Filter 2 Intra-frame 8 8 Prediction Output 4 3 Motion- Video 6 1 5 e.g., Mode 3: Compensation Signal Intra/Inter Diagonal down/right prediction a, f, k, p are predicted by Motion (A + 2Q + I + 2) >> 2 Data Motion Kasaei Estimation

Intra Intra- -Prediction Prediction 37 � Reducing spatial redundancies within a picture. Reducing spatial redundancies within a picture. � Prediction using surrounding pixels. � Used only in H.264. � 2 sizes for intra prediction. � 4x4 � 16x16

Luma 4x x4 Intra Modes 38 Kasaei

Luma 16x x16 Intra Modes 39 Kasaei

Optimal Intra Optimal Intra 4 4x x4 4 Mode Selection Mode Selection 40 � Selects the mode with the best R-D tradeoff. � Full search method: Divide each MB into sixteen 4x4 blocks. For each 4x4 block: For each of the nine lntra 4x4 prediction modes: For each of the nine lntra_4x4 prediction modes: � Predict the current 4x4 block by the current mode. � Get prediction residual. � Apply transform, quantization, entropy coding, inverse quantization, and inverse transform, find the output bits R, and the reconstruction error SAD or MSE. Compute the joint cost: SAD + λ (Q) R: C t th j i t t SAD λ (Q) R � Q: quantization step. � λ (Q) = 0.85 x 2 ^ (Q-12) / 3: obtained through experiments. � End Find the mode with the smallest cost as the best Intra_4x4 prediction mode _4 4 p for this 4x4 block. End � Fast method: An active research area. � Reduce the number of searches. Kasaei

Optimal Intra Optimal Intra 16 16x x16 16 Mode Selection Mode Selection 41 � Full search method: For each lntra l6x16 prediction mode: For each lntra_l6x16 prediction mode: � Get prediction of the current MB. � Find the prediction residual. � Perform 2D 4-point Hadamard transform for each 4x4 block. � Extract all the DC from the sixteen 4x4 blocks and apply 2D 4-point Hadamard � Extract all the DC from the sixteen 4x4 blocks and apply 2D 4-point Hadamard transform to the 4x4 DC again. � Cost estimation: Compute the absolute value of all the Hadamard transform coefficients. end Find the mode with the smallest cost as the best Intra_16x16 prediction mode for this MB. � Decision between Intra 4x4 and Intra 16x16: � Decision between Intra_4x4 and Intra_16x16: � Compare the costs of Intra_4x4 mode and Intra_16x16 mode to find the best mode. Kasaei

Inter Inter- -Prediction Prediction � Block size for inter-prediction: p � H.261 & MPEG-1: 16X16 � MPEG-2: 16X16, 16X8 � H.263 & MPEG-4: 16X16 & 8X8 3 4 � H.264: 16x16, 8x16, 16x8, 8x8, 4x8, 8x4, 4x4 � Motion estimation accuracy: � Motion estimation accuracy: � H.261: integer-pel � MPEG-1 & MPEG-2 & H.263: half-pel � MPEG-4 & H.264: quarter-pel MPEG 4 & H 264: quarter pel � Skip prediction 42

Motion Estimation (ME) Motion Estimation (ME) 43 � For each block, find the best match in the previous frame (reference frame) frame) � Upper-left corner of the block being encoded: (x0, y0) � Upper-left corner of the matched block in the reference frame: (x1, y1) � Motion vector (dx, dy): the offset of the two blocks: (dx, dy) = (x1 – x0, y1 – y0) (d d ) ( 1 0 1 0) � (x0, y0) + (dx, dy) = (x1, y1) � � Motion vector need to be sent to the decoder. Kasaei

Motion Compensation (MC) Motion Compensation (MC) 44 � Given reference frame and the motion vector, can obtain a prediction of the current frame. h f � Prediction error: Difference between the current frame and the predicted frame. � Prediction error will be coded by DCT, quantization, and entropy y , q , py coded. Kasaei

GOP, I, and P Frames GOP, I, and P Frames 45 � GOP: Group of pictures (frames). � I frames (Key frames): � Intra-coded frame, coded as a still image. � Can be decoded directly. � Used for GOP head, or at scene changes. � I frames also improve the error resilience. � P frames: (Inter-coded frames) � Predication-based coding, based on previous frames. Kasaei

GOP, I, P, and B Frames GOP, I, P, and B Frames 46 � B frames: Bi-directional interpolated prediction p p frames. � Predicted from both the previous frame and the next frame: more flexibilities -> better prediction. � Encoding order: 1 4 2 3 7 5 6. � Decoding order: 1 4 2 3 7 5 6. � Display order: 1 2 3 4 5 6 7 � Display order: 1 2 3 4 5 6 7. � Needs more buffers. � Needs buffer manipulations to p display the correct order. Kasaei

Block Matching Algorithms for ME Block Matching Algorithms for ME 47 � Split each frame into 16x16 blocks (MB), apply motion estimation for each macro-block each macro block. � Search window (maximum movement): w Typically 8, 16 or 32. � � Define a cost for finding the best match for each block in the previous frame. � Mean absolute error (MAE) or sum of absolute difference (SAD). � Mean squared error (MSE). � Sum of squared error (SSE). f q ( ) � Calculate the m otion vector (MV) between the current block and its counterpart in the previous frame. � Calculate the macro-block differences and send them. Kasaei

Cost Function Cost Function 48 � The best match is found by minimizing the sum of absolute difference (SAD) function absolute difference (SAD) function 16 6 , , 16 6 ∑ = − − − SAD ( s , c ( m )) s [ x , y ] c [ x m , y m ] x y = = x 1 , y 1 where s is the original video signal and c is the coded video signal video signal. Kasaei

Motion Estimation in H. Motion Estimation in H.264 264 49 � What is new? � Variable block size ME. � Can yield 15% bit rate savings. � Multiple reference frame ME. � 5-20% bit rate savings � 5-20% bit rate savings � Sub-pixel ME. p � 20% bit rate savings over integer ME. Kasaei

Variable Block Size ME Variable Block Size ME 50 Coder Coder Input Video Control Control Signal Data Transform/ Quant. Scal /Quant Scal./Quant. - Transf. coeffs Decoder Scaling & Inv. Split into Transform Macroblocks (16x16 pixels) ( p ) Entropy Entropy Coding De-blocking Filter16x16 8x8 16x8 8x16 Intra-frame MB 0 0 1 P Prediction di ti 0 0 1 Types 2 3 Output 1 Motion- Video 4x8 Compensation 8x8 8x4 4x4 Signal Intra/Inter 0 1 0 8x8 8x8 0 0 0 0 1 1 Motion Types 2 3 1 Data Motion Kasaei Estimation

Variable Block Size ME Variable Block Size ME - - Example Example 51 T=1 T=2 Kasaei

Variable Block Size ME Variable Block Size ME - - Example Example 52 T=1 T=2 Kasaei

Variable Block Size ME Variable Block Size ME - - Example Example 53 T=1 T=2 Kasaei

Multiple Reference Frames ME Multiple Reference Frames ME 54 Coder Coder Input Video Control Control Signal Data Transform/ Quant. Quant. S Scal./Quant. l /Q t - Transf. coeffs Decoder Scaling & Inv. Split into Transform Macroblocks (16x16 pixels) (16x16 pixels) Entropy Entropy Coding De-blocking Filter Intra-frame Prediction Output Motion- Video Compensation Signal Intra/Inter Multiple Reference Frames for Motion Data Motion Compensation Motion Kasaei Estimation

Sub- Sub -Pixel Motion Estimation Pixel Motion Estimation 55 When an object has a sub-pixel movement, the integer pixel ME cannot � describe it; so sub-pixel ME is defined. describe it; so sub-pixel ME is defined H.263 uses only half-pixel and MPEG-4 uses quarter-pixel accuracy � A gain of 1.5-2dB across the board over ½-pixel. � H.264 uses higher precision of spatial accuracy for ME up to eighth-pixel � accuracy. Kasaei

Search Window Search Window 56 � Search window (in previous frame) � Rectangle with the same coordinate as current block in current frame, extended by w pixels at each direction. q+2w q 2w w p+2w q q w w p w Kasaei

Full Search Method Full Search Method 57 � Full Search: � All candidates within search window are examined. � (2w+1) 2 positions should be examined. � Advantage: Good accuracy; finds the b best match. h � Disadvantage: Large amount of computation; (2w+1) 2 matches, 16x16 MAE for each match that is impractical for real time applications for real-time applications. � In order to avoid this complexity, we should reduce the search points so we have to use fast block matching algorithms algorithms. Kasaei

Mode Selection Mode Selection 58 � I t is not a normative part of the standard. � Due to variety of encoding modes for a MB, a mode selection should be made: = × × number b of f mod d es for f a MB MB number b of f QPs QP number b of f ref f pics i + × number of int er mod es number of QPs number of int ra mod es = × × + × + = 3 1 259 3 ( 9 4 ) 816 � Existing criteria: g S AD. � S ATD. � Rate-distortion optimization. �

Mode Selection Criteria Mode Selection Criteria • SAD: • SATD: 59

Mode Selection Criteria Mode Selection Criteria • Rate-Distortion Optimization: p min D ( X ) ≤ s . t . R ( ( X ) ) R T T Lagrange method: min D(x) + λ R(X) λ : trading off D and R 60

Mode Decision Method in H. Mode Decision Method in H.264 264/AVC /AVC 61 Calculate the RDCost for each Intra mode. � For each inter mode (16x16 16x8 8x16 and 8x8) For each inter mode (16x16, 16x8, 8x16 and 8x8), � � For each block in the current mode: � Do ME in a search area, select the point that minimizes below equation: � λ = + λ − J ( m , ) SAD ( s , c ( m )) . R ( m p ) motion motion End � C l Calculate the RDCost using: l h RDC i � RDCost = Distortion + λ ×Rate Note that: � Rate needs doing: Transform, Quantization, and entropy coding. � Distortion needs doing: Transform, Quantization Transform -1 , and Quantization -1 � End � From the calculated RDCosts: � (RDCost_Intra_16x16, RDCost_Intra_4x4, RDCost_I_PCM, RDCost_SKIP, (RDC t I t 16 16 RDC t I t 4 4 RDC t I PCM RDC t SKIP RDCost_Inter_16x16, RDCost_Inter_16x8, RDCost_Inter_8x16 and RDCost_Inter_8x8) S elect the least one as the best mode. � Kasaei

Slice Slice 62 � Each frame can be coded in one or more slices. Each containing one (16x16) or all the macro-blocks in the frame (1 slice per picture) Each containing one (16x16) or all the macro blocks in the frame (1 slice per picture). � � � Number of macro-blocks per slice need not be constant within a picture. � Because of minimal inter-dependency between coded slices the propagation of error can be limited. Kasaei

Slice Coding Slice Coding 63 � Slices can have different shapes and sizes. � Slices do not have to be consecutive in the raster scan. Sli d t h t b ti i th t � Each slice is self-contained. � Can be decoded without knowing the data of other slices. � Useful for: � Error resilience and concealment. � Parallel processing. Kasaei

Slice Type Slice Type 64 � Each slice can be coded as one of 5 types: � I slice: � All MBs are coded using intra mode. � P slice: � An MB can be coded in intra mode or inter mode with at most one prediction signal per block. � B slices: � In addition to modes in P slice some MBs can also be predicted using two � In addition to modes in P slice, some MBs can also be predicted using two prediction signal per block. � SP slice: Switching-P slice. � To facilitate switching between different video streams. g b � SI slice: Switching-I slice. � Using only Intra prediction. Kasaei

Transformation Transformation 65 � Transform Coding g � Reducing the spatial redundancy of prediction error � Like 2-D Fourier transform � Lossy Compression (not by itself only but) after quantization (ignoring y p ( y y ) q ( g g high frequency components) � Using integer transform instead of DCT � Similar properties with DCT � Integer operations � No worries about DCT and IDCT matching � Transform size � Other standards: 8x8 � H.264: 4x4, 8x8

Transformation Transformation 66 � H.264 uses three types of transforms: � Hadamard transform for 4x x4 array of luma DC coefficients. � Hadamard transform for 2x x2 array of chroma DC coefficients. � DCT-based transform for all other 4x4 blocks in residual data. 4 4 Kasaei

Transformation Transformation 67 Kasaei

Transformation Transformation 68 � Fundamental differences between Hadamard transform and DCT: � It is an integer transform. � It is possible to ensure zero mismatch between encoder and decoder. � Can be implemented using only additions and shifts. p g y � A scaling multiplication is integrated into the quantizer. � Can be carried out using 16-bit integer arithmetic. Kasaei

Transformation Transformation 69 � DCT: [1] Kasaei

Transformation Transformation 70 � DCT Approximation: pp [1] Kasaei

Transformation Transformation 71 � 4x x4 Hadamard transform: � 2x2 Hadamard transform: 2x2 Hadamard transform: [1] Kasaei

Quantization Quantization 72 � Requirements of complicated forward and inverse quantizers. � Avoid division and/or floating point arithmetic. � Incorporate the post- and pre-scaling matrices. � The basic forward quantizer: � A total of 52 values of Qstep are supported by the standard, indexed by a quantization parameter. ti ti t Kasaei

Quantization Quantization 73 � Wide range of quantizer step sizes makes it possible for an encoder to control the tradeoff accurately and flexibly between bit rate and quality. l h d ff l d fl ibl b bi d li � QP can be different for luma and chroma. Commonly QPC > QPY. � Kasaei

Quantization Quantization 74 � Forward quantization in integer arithmetic: For Intra mode: � For Inter mode: � MF: Multiplication Factor. � � Inverse quantization in integer arithmetic: Kasaei

Complete T & Q Complete T & Q 75 Kasaei

Deblocking Deblocking Filter Filter 76 � Applied to each decoded macro-block to reduce the blocking distortion. � Filtering is applied to vertical or horizontal edges of 4x x4 blocks in a macro-block (except for edges on slice boundaries). � Filter is stronger at places where there is likely to be significant blocking distortion. Such as the boundary of an Intra coded macro-block or a boundary between blocks that � contain coded coefficients. � Effect of the filter decision is to switch off the filter when there is a significant change (gradient) across the block boundary in the original image image. Kasaei

Deblocking Deblocking Filter Filter 77 � Only in H.26x standards. Only in H.26x standards. With d bl With deblocking filter. ki filt Without deblocking filter.

Error Resilience Tools Error Resilience Tools 78 � Error Resilience Tools: � To minimize the visual effect of error within a frame. � To avoid error propagation. � These tools include: Flexible macro-block ordering (FMO). 1. Arbitrary slice ordering (ASO). 2. Redundant slices (RS). 3. Slice data partiti oning (DP). Slice data partiti oning (DP) 4 4. Slice structured coding. 5. Flexible reference frame concept. 6. Picture switching. g 7. Intra-coding. 8. Kasaei

Flexible Macro Flexible Macro- -Block Order Block Order 79 � FMO can work to randomize the data prior to transmission. So that if a segment of data is lost, the errors are distributed more randomly over the video S th t if t f d t i l t th di t ib t d d l th id � � pictures. � Relevant neighboring data is available for concealment of lost content. g g Kasaei

Arbitrary Slice Order Arbitrary Slice Order 80 � ASO allows slices of a picture to appear in any order for delay reduction. Particularly for use on networks that can deliver data packets out of order. � Kasaei

Redundant Slices Redundant Slices 81 � Coding with a low QP (and hence in good quality), the RS is coded with a high QP (utilizing fewer bits). hi h QP ( ili i f bi ) � Decoder decodes primary slice, if it is available, and discards the RS. � If the primary slice is missing, the RS can be reconstructed. Kasaei

Data Partitioning Data Partitioning 82 � Type A: Header information. Including MB types, quantization parameters, and motion vectors. � � Type B: Intra Partition. T B I t P titi It carries Intra CBPs and Intra coefficients. � � Type C: Inter Partition � Type C: Inter Partition. It contains only Inter CBPs and Inter coefficients. � Kasaei

Baseline Profile Baseline Profile 83 � Baseline: Progressive, Videoconferencing & Wireless. � I and P picture types (not B). I d P i ( B) Uses list0 for P-Slices. � In-loop deblocking filter. � 1/4-sample motion compensation. 1/4 l ti ti � Tree-structured motion segmentation down to 4x4 block size. � VLC-based entropy coding (UVLC and CAVLC). � Some enhanced error resilience features. S h d ili f t � Flexible macro-block ordering/arbitrary slice ordering. � Redundant slices. Kasaei

Main Main Profile Profile 84 � May use list0 and/or list1 for B-Slices. � B-Slices may use: � One past and one future reference. p � Two past references. � Two future references. Kasaei

B- -Slice Prediction Slice Prediction 85 Kasaei

Extended Extended Profile Profile 86 � SI and SP slices are specially coded slices that enable (among other things) efficient s itching bet een ideo streams and efficient random access for ideo efficient switching between video streams and efficient random access for video decoders. � A common requirement in a streaming application is for a video decoder to � A common requirement in a streaming application is for a video decoder to switch between one of several encoded streams. � SI slices are the same as SP Slices but their prediction is Intra 4x4 from � SI slices are the same as SP Slices, but their prediction is Intra 4x4 from the previously decoded and reconstructed image samples. Kasaei

Switching Switching 87 Kasaei

SP Slice SP Slice 88 Kasaei

Random Access to Video Frames Random Access to Video Frames 89 � Can decode A0, create SP-Slice A0-10, and predict A11 from A0. Kasaei

H. H.264 264/AVC Extension ( /AVC Extension (Frext Frext) ) 90 � The first version of the standard uses: � The 4:2:0 chroma format. � Typically derived by performing an RGB-to-YCbCr color-space transformation. � 8 bit sample precision for luma and chroma values. Kasaei

FRExt FRExt Color Space Color Space 91 � The FRExt amendment extended the standard to 4:2:2 and 4:4:4 chroma formats and higher than 8 bits precision. h f d hi h h 8 bi i i � In 4:2:0 chroma format, each macro-block consists of a 16x16 region of luma samples and two corresponding 8x8 chroma sample arrays. � In a macro-block of 4:2:2 chroma format video, the chroma sample 4 , p arrays are 8x16 in size; and in a macro-block of 4:4:4 chroma format video, they are 16x16 in size. � Frext uses YCgCo the color space (where the "Cg" stands for green chroma and the "Co" stands for orange chroma), which is much simpler and typically has equal or better coding efficiency. yp y q g y Kasaei

FRExt FRExt Color Space Color Space 92 Kasaei

Scalable Video Coding Extension Scalable Video Coding Extension of H. of H.264 of H. of H.264 264/AVC 264/AVC /AVC /AVC

History of SVC History of SVC 94 � Hybrid video coding: y g � Motion compensated DPCM + spatial decorrelating transformations. � Difference between encoder and decoder prediction loop Diff b d d d d di i l leads to the drift problem. � Thus, video coding techniques based on motion- compensated 3-D wavelet transform have been compensated 3 D wavelet transform have been extensively studied. Kasaei

History of SVC (Cont.) History of SVC (Cont.) 95 � MPEG issued a call for proposals for efficient p p scalable video coding technology in October 2003. � 12 of 14 submitted proposals were based on 3-D wavelet transforms, the other two were extensions of H H.264/AVC. 6 /AVC Kasaei

History of SVC (Cont.) History of SVC (Cont.) 96 � After six months of extensive study, the scalable y, extension of H.264/AVC was chosen as the starting point of MPEG’s SVC project in October 2004. � In January 2005, MPEG and VECG agreed to jointly finalize the SVC project as an Amendment of fi li th SVC j t A d t f H.264/AVC within the joint video team. Kasaei

Types of Scalability in SVC Types of Scalability in SVC 97 � Temporal Scalability. p y � Spatial scalability. � Quality scalability. Q y y � Coarse grain scalability. � Fine grain scalability. Kasaei

Temporal Scalability Temporal Scalability 98 � Temporal layers. p y � MCP is restricted to reference pictures of the lower temporal layer. � Hierarchical B-pictures. � In general, hierarchical prediction structures can be combined with the multiple reference picture concept of H.264/AVC. Kasaei

Temporal Scalability Temporal Scalability 99 Kasaei

Temporal Scalability Temporal Scalability 100 � It is possible to arbitrarily adjust the structural delay p y j y between encoding and decoding of a picture by restricting MCP from that follow the picture to be predicted display order (Fig. 1c). di t d di l d (Fi ) � However, the coding efficiency typically decreases. H th di ffi i t i ll d Kasaei