Image and Video Coding: Video Coding Extensions Screen Content - PowerPoint PPT Presentation

Image and Video Coding: Video Coding Extensions

Screen Content Coding Screen Content Coding sensor-captured video content screen content video Screen Content Video Increasingly becoming important for a number of applications (e.g., online meetings) Screen content video sequences have different properties than sensor-captured video sequences Coding efficiency could be improved by dedicated coding tools / coding modes Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 2 / 34

Screen Content Coding / Coding Tools Transform Skip Mode Quant. DCT-II IDCT-II Dequant. Transform Coding Efficiency for Screen Content Less energy compaction as for typical sensor-captured content Strong quantization can result in disturbing artefacts Transform Skip Mode Coding mode for which no transform is carried out (indicated by special flag) Direct quantization of residual samples Can be combined with dedicated entropy coding for quantization indexes Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 3 / 34

Screen Content Coding / Coding Tools Block Differential Pulse Code Modulation (BDPCM) no BDPCM horizontal BDPCM vertical BDPCM prediction quantization q [ x , y ] = 0 ˆ q [ x , y ] = q [ x − 1 , y ] ˆ q [ x , y ] = q [ x , y − 1 ] ˆ Exploit Dependencies in Transform Skip Mode Quantization indexes are not directly transmitted by entropy coding Two additional modes for prediction of quantization indexes (inside block): Horizontal prediction (first column is not predicted) Vertical prediction (first row is not predicted) Entropy coding of prediction errors ∆ q = q [ x , y ] − ˆ q [ x , y ] Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 4 / 34

Screen Content Coding / Coding Tools Intra Block Copy restrictions in VVC valid 64 × 64 region invalid 64 × 64 region curr curr curr curr “Motion-compensated” prediction inside a picture with integer-sample accurate motion vectors To reduce memory access complexity, VVC includes restrictions of permitted motion vectors Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 5 / 34

Screen Content Coding / Coding Tools Palette Mode run = 5 palette 0 0 0 0 0 0 0 0 0 1 0 1 1 1 1 2 1 1 1 1 1 1 G|Y B|Cb R|Cr 0 1 1 1 1 1 1 4 index = 1, run = 5 y 0 x 0 z 0 0 0 0 0 1 1 2 2 2 y 1 1 x 1 z 1 run = 7 0 3 3 1 1 2 2 2 x 2 y 2 z 2 2 0 3 3 1 1 2 2 2 0 3 3 1 1 2 2 2 x 3 y 3 z 3 3 0 3 3 3 4 2 2 2 0 3 3 1 1 2 2 2 4 escape 0 0 0 0 0 0 0 0 copy above, run = 7 Alternative Coding Mode: Palette Mode Quantized color vectors are represented by palette indexes Palette for current block is predictively coded referring to preceding palettes Palette can include an escape symbol for representing less likely values Palette indexes are coded using horizontal or vertical scanning, using two coding modes 1 Index mode: Transmit palette index and run length ( ≥ 0) 2 Copy mode: Index is copied from top (hor. scan) or left (ver. scan), transmit run length ( ≥ 0) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 6 / 34

Screen Content Coding / Coding Efficiency Coding Efficiency Example: ”Desktop” (1920 x 1080) PSNR [dB] 42 VVC with additional 41 screen content tools 40 VVC without 39 screen content tools 38 37 36 35 34 33 32 31 30 29 28 bit rate [Mbits/s] 27 0 0 . 5 1 1 . 5 2 2 . 5 3 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 7 / 34

Screen Content Coding / Coding Efficiency Subjective Comparison: “Desktop” (Crop of Top-Left Region) VVC without SCC tools @ 1 Mbit/s VVC with SCC tools @ 1 Mbit/s Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 8 / 34

Screen Content Coding / Coding Efficiency Coding Efficiency Impact of Screen Content Coding Tools (Example: VVC) average bit-rate savings intra only random access low delay ChineseEditing 38 % 36 % 32 % Console 66 % 52 % 48 % Desktop 67 % 61 % 57 % FlyingGraphics 41 % 18 % 14 % SlideEditing 47 % 44 % 36 % SlideShow 20 % 16 % 10 % average 46 % 38 % 33 % Average Bit Rate Savings Bit-rate savings based on PSNR as quality measure Averages over reasonable quality range Screen content tools provide large gains for many sequences Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 9 / 34

Scalable Video Coding / Types of Scalability Scalable Video Coding bitstream video video encoder decoder original: 1080p, 60Hz 1080p, 60Hz, 10 MBits/s Scalable Bitstream video decoder Includes multiple coded versions of a video sequence 1080p, 60Hz, 5 MBits/s Representations must be extractable by simple discarding of packets Decoder or middlebox can extract video decoder representation suitable for application requirements 720p, 30Hz, 1.5 MBits/s Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 10 / 34

Scalable Video Coding / Types of Scalability Types of Scalability Temporal Scalability Scalable bitstream contains representations with different frame rates Spatial Scalability Scalable bitstream contains representations with different spatial resolutions Quality Scalability Scalable bitstream contains representations with different bit rates (but same resolution) Combined Scalability Combination of two or more of the above types Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 11 / 34

Scalable Video Coding / Temporal Scalability Temporal Scalability 0 3 2 4 6 1 base layer I B B B B 4 5 7 8 additional enhancement B B B B layer pictures Coding Structures for Temporal Scalability Requirement: Enhancement layer picture are not used for prediction of base layer pictures Hierarchical B picture are well suited and provide very high coding efficiency Very small loss in coding efficiency relative to best possible single layer coding Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 12 / 34

Scalable Video Coding / Quality and Spatial Scalability Quality / SNR Scalability 1 9 7 11 5 15 13 17 3 enhancement layer B I B B B B B B B B 0 8 6 10 4 14 12 16 2 base layer I B B B B B B B B Inter-Layer Prediction Add co-located base layer picture to reference list of enhancement layer picture Base layer data are exploited by sample prediction and motion prediction Improves coding efficiency relative to independent coding of both layers (simulcast) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 13 / 34

Scalable Video Coding / Quality and Spatial Scalability Spatial Scalability 1 9 7 11 5 15 13 17 3 enhancement layer B I B B B B B B B B upsampler upsampler upsampler upsampler upsampler upsampler upsampler upsampler upsampler 0 8 6 10 4 14 12 16 2 base layer I B B B B B B B B Inter-Layer Prediction with Upsampling Add upsampled co-located base layer picture to reference list of enhancement layer picture Use information coded in base layer for improving coding efficiency relative to simulcast Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 14 / 34

Scalable Video Coding / Quality and Spatial Scalability Multi-Layer and Combined Scalability 2 13 11 15 8 20 18 22 5 layer 2 B B B B B B B B B 1 12 10 14 7 19 17 21 4 layer 1 B B B B B B B B B 0 9 6 16 3 layer 0 I B B B B Multiple quality and/or spatial enhancement layers are possible Coding efficiency for top layer decreases with number of supported layers Decoding complexity for top layer increases with number of supported layers Temporal scalability can be straightforwardly combined with quality/spatial scalability Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 15 / 34

Multiview and 3D Video Coding / Stereo and Multiview Coding 3D Cinema / Home Cinema: Stereo Video display positive parallax Why Glasses ? Need to project different image to each eye Glasses control over what each eye sees Need to transit video with two images per time instance Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 16 / 34

Multiview and 3D Video Coding / Stereo and Multiview Coding Stereo Video Example Similarities between left and right picture for same time instance Can be exploited by technique similar to motion-compensated prediction Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 17 / 34

Multiview and 3D Video Coding / Stereo and Multiview Coding Multi-view Coding with Disparity-Compensated Prediction 0 5 4 6 2 left view (primary) I B B B B 1 6 5 7 3 right view (secondary) B B B B B Multiview Coding with Disparity-Compensated Prediction Add reconstructed picture of primary view to reference lists for secondary view (same time instance) Only change required is construction of reference picture lists Straightforward extension to more than 2 views Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Video Coding Extensions 18 / 34