Image and Video Coding: Introduction bitstream encoder decoder - PowerPoint PPT Presentation

Image and Video Coding: Introduction bitstream encoder decoder

Motivation Image and Video Coding video bitstream bitstream video transmission encoder decoder or storage data data Main Goal of Image and Video Coding Efficient transmission or storage of images and videos Reduce the bit rate for a given amount of video data Image and Video Coding: Enabling Technology Enables new application or makes them economically feasible Distribution and storage of images and video Digital television Video streaming Video conferencing Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 2 / 48

Motivation Important Image and Video Coding Standards of ITU-T and ISO/IEC JPEG (1992) [ ISO/IEC 10918-1 | ITU-T Rec. T.81 ] Storage and distribution of digital images MPEG-2 Video (1995) [ ITU-T Rec. H.262 | ISO/IEC 13818-2 ] Standard definition (SD): Storage (DVD-Video) and digital television broadcast (DVB-T) H.264 | AVC : Advanced Video Coding (2003) [ ITU-T Rec. H.264 | ISO/IEC 14496-10 ] High definition (HD): Storage (Blu-ray) and digital television broadcast (DVB-S) Video streaming H.265 | HEVC : High Efficiency Video Coding (2013) [ ITU-T Rec. H.265 | ISO/IEC 23008-2 ] Ultra-high definition (UHD) video storage (UHD Blu-ray), Image storage (BPG, HEIF) Broadcast (HD: DVB-T2, UHD: DVB-S2), UHD video streaming Current project: H.266 | VVC : Versatile Video Coding (to be finalized in July 2020) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 3 / 48

Motivation Estimated Global Internet and Video Traffic global Internet traffic Internet video traffic Exabytes 300 per month 250 200 150 82% 100 50 73% 0 2016 2017 2018 2019 2020 2021 [ Cisco: “The Zettabyte Era: Trends and Analysis”, 2017 ] Still Need Better Video Coding Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 4 / 48

Source Data for Image and Video Coding / Gray-Level Images Single-Component Image Matrix of integer samples x = 0 , 1 , · · · , W − 1 s [ x , y ] with y = 0 , 1 , · · · , H − 1 Each sample can take values in a given range 0 , 2 B − 1 � � s [ x , y ] ∈ with B = bit depth x Single-component image is characterized by Image width W H Image height H y Sample bit depth B Number of bits for raw image data N bits = W · H · B W Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 5 / 48

Source Data for Image and Video Coding / Gray-Level Images Gray-Level Image Example: Impact of Image Size (Spatial Resolution) 400 × 300 samples 200 × 150 samples 100 × 75 samples 50 × 38 samples Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 6 / 48

Source Data for Image and Video Coding / Gray-Level Images Gray-Level Image Example: Impact of Image Size (Spatial Resolution) 400 × 300 samples 200 × 150 samples � interpolated � to 400 × 300 100 × 75 samples 50 × 38 samples � interpolated � interpolated � � to 400 × 300 to 400 × 300 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 6 / 48

Source Data for Image and Video Coding / Gray-Level Images Gray-Level Image Example: Impact of Sample Bit Depth 8 bits per 6 bits per sample sample 4 bits per 2 bits per sample sample Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 7 / 48

Source Data for Image and Video Coding / Color Images Color Images red, green, blue sample values Human eye has three types of color receptors Require three color components Cameras/displays typically use red, green, blue A color image is represented as three matrices of samples (one for each color component) Number of bits for raw image data N bits = 3 · W · H · B Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 8 / 48

Source Data for Image and Video Coding / Color Images YCC Color Formats RGB (red,green,blue) format unsuitable for compression: Large amount of statistical dependencies YCC formats: Transform samples at the same spatial position ( x , y )           Y [ x , y ] R [ x , y ] 0  = round  +  M 3 × 3 2 B − 1 RGB �→ YCC : C 1 [ x , y ] G [ x , y ]  ·       2 B − 1 C 2 [ x , y ] B [ x , y ]             R [ x , y ] Y [ x , y ] 0  = round  M − 1 2 B − 1 YCC �→ RGB : G [ x , y ] C 1 [ x , y ]  −  ·   3 × 3       2 B − 1 B [ x , y ] C 2 [ x , y ] (transform matrix M 3 × 3 depends on actual YCC format and RGB color space) Most common format in image and video coding: YCbCr format Y : Luma component (representing brightness) Cb : Scaled difference between blue and luma Cr : Scaled difference between red and luma Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 9 / 48

Source Data for Image and Video Coding / Color Images Example: Comparison of RGB and YCbCr format sample values RGB YCbCr sample values Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 10 / 48

Source Data for Image and Video Coding / Color Images Color Sampling Formats RGB YCbCr 4:4:4 YCbCr 4:2:2 YCbCr 4:2:0 most common color format � half the number � of RGB samples Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 11 / 48

Source Data for Image and Video Coding / Videos Video: Sequence of Images Video Sequence of images Characterized by Image size W × H Sample bit depth B Color format (typically, YCbCr 4:2:0) Frame rate F (pictures per second) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 12 / 48

Source Data for Image and Video Coding / Videos Video Example: Impact of Frame Rate F = 50 Hz ( 50 pictures per second ) F = 5 Hz ( 5 pictures per second ) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 13 / 48

Source Data for Image and Video Coding / Raw Data Rate Raw Video Data Rate Raw Video Data Rate: Bit rate of raw video data R raw = ( samples per time unit ) · ( bit depth per sample ) = ( frame rate F ) · ( image size W · H ) · ( color format factor C ) · ( bit depth B ) Example 1: Ultra High Definition (UHD) Video 60 pictures per second (US, Japan) 3840 × 2160 luma samples, YCbCr 4:2:0 color format, 10 bits per sample raw data rate : R raw = 60 Hz · 3840 · 2160 · ( 3 / 2 ) · 10 bits ≈ 7 . 5 Gbits/s two-hour video: file size ≈ 6.7 TByte (without audio) Example 2: Full HD Lecture Video 60 pictures per second 1920 × 1080 luma samples (full HD), RGB color format (screen capture), 8 bits per sample raw data rate : R raw = 60 Hz · 1920 · 1080 · 3 · 8 bits ≈ 2 . 986 Gbits/s 90 min. lecture: file size = 2 . 986 Gbits/s · 90 min · 60 s/min / ( 8 bit/byte ) ≈ 2 TByte Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 14 / 48

The Image and Video Coding Problem / Video Communication Typical Video Communication Scenario pre- processing capture video video encoder encoder raw input raw input data samples data samples encoded bitstream transmission channel (can be replaced by storage) bitstream channel channel no transmission errors demodulator channel modulator decoder encoder received bitstream output and perception video video decoder decoder raw output raw output post- data samples data samples processing Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 15 / 48

The Image and Video Coding Problem / Video Communication Typical Video Communication Scenario pre- processing capture video video encoder encoder raw input raw input data samples data samples encoded bitstream transmission channel (can be replaced by storage) bitstream channel channel no transmission errors demodulator channel modulator decoder encoder received bitstream output and perception video video decoder decoder raw output raw output post- data samples data samples processing Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 15 / 48

The Image and Video Coding Problem / Video Communication Typical Video Communication Scenario pre- processing capture video video encoder encoder raw input raw input data samples data samples encoded bitstream transmission channel (can be replaced by storage) bitstream channel channel no transmission errors demodulator channel modulator decoder encoder received bitstream output and perception video video decoder decoder raw output raw output post- data samples data samples processing Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 15 / 48