Image and Video Coding: Representation, Acquisition, Display ... - PowerPoint PPT Presentation

Image and Video Coding: Representation, Acquisition, Display ... 10011 ... encoder decoder

Representation Formats Representation Formats ... 10011 ... encoder decoder representation format bitstream representation format Raw Data Formats for Exchanging Pictures and Videos Output of camera, input to video encoder, output of video decoder, input to display Examples: BT.601 (SD), BT.709 (HD), BT.2020 (UHD) Images and Videos : Sample arrays characterized by Spatio-temporal sampling, linear color space (color gamut) Non-linear encoding (transfer function), color representation format (RGB, YCbCr, ...) Quantization (bit depth) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 2 / 43

Representation Formats / Spatio-Temporal Sampling Spatio-Temporal Sampling Discrete Representation of Continuous Irradiance Pattern on Image Sensor Each image is represented by a W × H array of samples c n [ ℓ, m ] = c cont ( ℓ · ∆ x , m · ∆ y , n · ∆ t ) Spatial sampling is typically done by image sensor (photocells of finite size) Video: Multiple pictures taken per second (for example: 50 per second) Gray-Level Image Color image 2D array of samples Three 2D arrays of samples (one per color components) x y Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 3 / 43

Representation Formats / Spatio-Temporal Sampling Spatial Sampling Orthogonal Progressive Sampling W · ∆ x Image width W and image height H x Sample aspect ratio (SAR) SAR = ∆ x ∆ x ∆ y H · ∆ y ∆ y y Picture aspect ratio (PAR) PAR = W · ∆ x H · ∆ y = W H · SAR Interlaced Sampling (old special case) top field bottom field Top field: Even scan lines top field Bottom field: Odd scan lines bottom field top field Top and bottom fields are alternatively bottom field scanned at successive time instances Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 4 / 43

Representation Formats / Spatio-Temporal Sampling Common Picture Formats picture size sample aspect picture aspect (in samples) ratio (SAR) ratio (PAR) 720 × 576 12 : 11 4 : 3 standard 720 × 480 10 : 11 4 : 3 definition 720 × 576 16 : 11 16 : 9 720 × 480 40 : 33 16 : 9 1280 × 720 1 : 1 16 : 9 high 1440 × 1080 4 : 3 16 : 9 definition 1920 × 1080 1 : 1 16 : 9 ultra-high 3840 × 2160 1 : 1 16 : 9 definition 7680 × 4320 1 : 1 16 : 9 Note: In SD formats, only 704 samples are displayed per line (overscan) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 5 / 43

Representation Formats / Spatio-Temporal Sampling Illustration: Spatial Resolution, Sample Aspect Ratio, Picture Aspect Ratio CIF (352 × 288), SAR 12 : 11 CIF (352 × 288), SAR 16 : 11 QCIF (176 × 144), SAR 16 : 11 on display (512 × 288); PAR 4 : 3 on display (512 × 288); PAR 16 : 9 on display (512 × 288); PAR 16 : 9 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 6 / 43

Representation Formats / Color Space Representation of Color Images Color Components Trichromatic vision: Require 3 color components Representation formats are display-oriented: Linear RGB color space with real primaries Require conversion from camera color space to RGB color space of the representation format     R R G = M 3 × 3 · G     B B rep. format camera Conversion matrix M 3 × 3 typically includes white balancing Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 7 / 43

Representation Formats / Color Space Color Gamut RGB color space: Specified by chromaticity coordinates of primaries and the white point The chosen RGB color space determines the representable color gamut y human gamut ProPhoto RGB BT.709 BT.2020 ProPhoto XYZ BT.2020 (UHD) x r 0.6400 0.7080 0.7347 1.0000 red [ wide color gamut ] y r 0.3300 0.2920 0.2653 0.0000 x g 0.3000 0.1700 0.1596 0.0000 green 0.6000 0.7970 0.8404 1.0000 y g BT.709 (HD) 0.1500 0.1310 0.0366 0.0000 x b blue y b 0.0600 0.0460 0.0001 0.0000 D65 white x w 0.3127 0.3127 0.3457 0.3333 white y w 0.3290 0.3290 0.3585 0.3333 XYZ x Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 8 / 43

Representation Formats / Color Space Color Space Conversion RGB color spaces are linearly related to XYZ color space (and cone excitation space LMS) XYZ color space is given by color-matching functions ¯ x ( λ ) , ¯ y ( λ ) , ¯ z ( λ ) Determination of Conversion Matrix to XYZ space        X w / Y w     1  X X r X g X b R X r X g X b white point 1 1 Y Y r Y g Y b G Y r Y g Y b  · − − − − − − − − →  ·  =  =         Z Z r Z g Z b B Z w / Y w Z r Z g Z b 1 Use X / Y = x / y and Z / Y = ( 1 − x − y ) / y x g  x r x b  x w     Y r y r y g y b y w 1  = 1 1 1 Y g        1 − x g − y g 1 − x w − y w 1 − x r − y r 1 − x b − y b Y b y w y r y g y b Solve linear equation system for unknown values Y r , Y g , and Y b Determine resulting matrix (using X = x / y · Y and Z = ( 1 − x − y ) / y · Y ) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 9 / 43

Representation Formats / Color Space Example: Conversion between XYZ and sRGB (BT.709) red green blue white x 0.6400 0.3000 0.1500 0.3127 y 0.3300 0.6000 0.0600 0.3290 Linear equation system (only 8 digits of precision shown)  0 . 95045593   1 . 93939394 0 . 5 2 . 5    Y r 1  = 1 1 1 Y g      1 . 08905775 0 . 09090909 0 . 16666667 13 . 16666667 Y b Solution (6 digits of precision shown) Y r = 0 . 212673 , Y g = 0 . 715152 , Y b = 0 . 072175 Resulting conversion matrix (4 digits of precision)       0 . 4125 0 . 3576 0 . 1804 X R Y  = 0 . 2127 0 . 7152 0 . 0722 G      Z 0 . 0193 0 . 1192 0 . 9503 B sRGB Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 10 / 43

Representation Formats / Non-Linear Encoding Non-Linear Encoding / Gamma Encoding Human Vision: Non-Linear Brightness Perception Weber-Fechner law: Perceivable difference in luminance depends on background luminance Certain amount of quantization noise is more visible in dark image regions Reduce effect by non-linear encoding components before quantization E ′ = f TC ( E ) Gamma Encoding / Gamma Decoding Encoder side: Approximation by power law Y ′ = f TC ( Y ) = Y γ e with encoding gamma γ e ≈ 1 / 2 . 2 ≈ 0 . 45 with Y being the relative luminance in range [ 0 ; 1 ] Decoder side: Invert the gamma encoding Y = f − 1 TC ( Y ′ ) = ( Y ′ ) γ d with decoding gamma γ d ≈ 1 /γ e ≈ 2 . 2 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 11 / 43

Representation Formats / Non-Linear Encoding Transfer Characteristics 1 γ e = 1 / 2.2 linear increasing Y non-linear encoded signal E' 0.8 BT.709 0.6 BT.2020 0.4 linear increasing Y ′ = f TC ( Y ) 0.2 linear encoding 0 0 0.2 0.4 0.6 0.8 1 linear component signal E Representation Formats Piecewise-defined transfer function (linear function for very small values) � κ · E : 0 ≤ E < b E ′ = f TC ( E ) = a · E γ − ( a − 1 ) : b ≤ E ≤ 1 BT.709 / BT.2020 : γ = 0 . 45, κ = 4 . 5, a ≈ 1 . 0993, b ≈ 0 . 0181 High Dynamic Range (HDR) : Modified transfer functions (PQ, HLG) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 12 / 43

Representation Formats / YCC Color Formats YCC Color Formats Reasons for using YCC color formats Color television: Add color difference information to black & white television Decorrelation of color components (remember: opponent processes) Luminance-related signal L Two color difference signals C 1 , C 2 (e.g., yellow-blue & red-green) Consider mapping LC 1 C 2 �→ RGB �→ XYZ         X X r X g X b R ℓ R c 1 R c 2 L  =  ·  · Y Y r Y g Y b G ℓ G c 1 G c 2 C 1      Z Z r Z g Z b B ℓ B c 1 B c 2 C 2 Desirable properties 1 Achromatic signals ( x = x w and y = y w ) have C 1 = C 2 = 0 2 Changes in C 1 and C 2 do not have any impact on relative luminance Y Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 13 / 43

Representation Formats / YCC Color Formats / The YCbCr Format YCbCr: Achromatic Signals have Zero Color Differences         X X r X g X b R ℓ R c 1 R c 2 L  =  ·  · Y Y r Y g Y b G ℓ G c 1 G c 2 C 1      Z Z r Z g Z b B ℓ B c 1 B c 2 C 2 Fulfilling desirable properties 1 Achromatic signals ( x = x w and y = y w ) have C 1 = C 2 = 0 In the RGB format, this means C 1 = C 2 = 0 = R = G = B (white/gray point) ⇒ Hence, we require R ℓ = G ℓ = B ℓ This choice yields     R ℓ R c 1 R c 2 a ? ?  = G ℓ G c 1 G c 2 a ? ?    B ℓ B c 1 B c 2 a ? ? Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Representation, Acquisition, Display 14 / 43