Image and Video Coding: Human Visual Perception Y 1 X 1 1 Z - PowerPoint PPT Presentation

Image and Video Coding: Human Visual Perception Y 1 X 1 1 Z

The Human Eye / Structure of the Human Eye The Human Eye Human Eye : Similar components as a camera Two-lens system: Cornea and crystalline lens Variable aperture: Pupil / iris Light-sensitive surface: Retina Data processing: Visual cortex of the brain Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 2 / 58

The Human Eye / Human Photoreceptors Light-Sensitive Cells in Retina Rods About 100 million rods in retina (each eye) More light-sensitive than cone cells ( ≈ 100 times) Responsible for scotopic vision (night vision) Low visual acuity, no color perception Cones About 5 million cones in retina (each eye) Less light-sensitive than rod cells Responsible for photopic vision (well-lit conditions) High visual acuity Colorized picture of rods (yellow) and cones (blue) as seen through a scanning Responsible for color vision electron microscope Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 3 / 58

The Human Eye / Human Photoreceptors Distribution of Rods and Cones 20 20 receptor density [10 4 / mm 2 ] 18 18 blind spot cones 16 16 rods 14 14 12 12 10 10 8 8 6 6 4 4 nasal retina temporal retina 2 2 0 0 -60 -60 -40 -40 -20 -20 0 0 20 20 40 40 visual angle relative to center of fovea [degree] Scotopic vision mainly at outer parts of the retina Photopic and color vision centered at the fovea (by far highest visual acuity) No receptors at blind spot (optic nerve) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 4 / 58

The Human Eye / Spectral Sensitivity Spectral Sensitivity of Human Vision 1 luminous efficiency 0.8 photopic vision 0.6 scotopic vision 0.4 0.2 0 400 450 500 550 600 650 700 750 wavelength λ [nm] Luminous Efficiency Functions Characterize spectral sensitivity of brightness perception Experimentally determined in brightness matching experiments V ( λ ) for photopic vision (standardized in 1924), V ′ ( λ ) for scotopic vision (standardized in 1951) Visible light : Electromagnetic radiation in spectrum 390 nm − 700 nm Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 5 / 58

The Human Eye / Spectral Sensitivity Radiometric vs Photometric Quantities Two parallel systems of quantities for characterizing electromagnetic radiation Radiometric quantities : Unweighted absolute energy / power measures Photometric quantities : Every wavelength is weighted with luminous efficiency function V ( λ ) Example : Radiance Φ – Luminance I [ unit: W / ( sr · m 3 ) ] Given is a radiance spectrum Φ( λ ) [ unit: W / ( sr · m 2 ) ] Radiometric quantity: Radiance � ∞ Φ = Φ( λ ) d λ 0 [ unit: cd / m 2 ] Equivalent photometric quantity: Luminance � ∞ K = 683 cd · sr I = K V ( λ ) · Φ( λ ) d λ with W 0 Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 6 / 58

The Human Eye / Spectral Sensitivity Trichromatic Vision Three Cone Types : Postulated by Young and Helmholtz (confirmed later by measurements) Responsible for color vision under photopic conditions S-cones (S for short wavelength) Most sensitive to blue light ( ≈ 430 nm) Only 5-6% of the cones are S-cones (blue light is blurred due to chromatic aberration) M-cones (M for medium wavelength) Most sensitive to green light ( ≈ 530 nm) L-cones (L for long wavelength) Most sensitive to yellow-green light ( ≈ 560 nm) colorized picture of cone mosaic in central region of fovea Ratio of L/M cones highly varies for different individuals Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 7 / 58

The Human Eye / Spectral Sensitivity Spectral Sensitivity of Human Cones 1 normalized sensitivity S-cones 0.8 rods 0.6 M-cones 0.4 L-cones 0.2 0 400 450 500 550 600 650 700 750 wavelength λ [nm] Cone fundamentals Spectral sensitivities with respect to light entering the cornea Estimated by comparing color-matching data (discussed later) of individuals with normal vision and individuals lacking one or two cone types Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 8 / 58

The Human Eye / Comparison to Other Species Color Perception of Other Species Monochromatic vision : Seals, sea lions, walruses, dolphins, whales Dichromatic vision : Most mammals (except sea mammals and some primates) Trichromatic vision : Humans and closely related primates, also bees Tetrachromatic vision : Many species of birds, fish, amphibians, reptiles, arachnids, and insects Pentachromatic vision : Some butterflies and birds (e.g., pigeons) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 9 / 58

The Human Eye / Comparison to Other Species The King of Color Vision: The Mantis Shrimp 16 different types of photoreceptors Can detect polarized light Depth vision with a single eye (3 focal points per eye) [ Marshall, et al, 2007 ] Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 10 / 58

The Human Eye / Luminance Sensitivity Luminance Sensitivity Sensing Capabilities of Human Eye From 10 − 6 cd / m 2 (visual threshold) to 10 5 cd / m 2 (sunny day) In each moment, only 2-3 orders of magnitude Adaptation to Lighting Conditions Pupillary light reflex: Fast but rather small effect Main factor: Photochemical reactions in the pigments of rods and cones Sensitivities of cones are nearly independently adjusted Weber-Fechner Law Ability to distinguish areas with different luminances depends on background brightness For a wide range of luminance levels I ∆ I ≈ const (approx. 1-2%) I Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 11 / 58

The Human Eye / Neural Processing in Retina Opponent Colors 1 yellow-blue achromatic normalized sensitivity 0.5 0 -0.5 red-green -1 400 450 500 550 600 650 700 750 wavelength λ [nm] Hering (1920): Colors never look reddish-green or yellowish-blue Neurons in retina transform cone responses into opponent signals (effective decorrelation): Achromatic signal + red-green difference + yellow-blue difference Opponent signals could be measured in stimulated retina tissue Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 12 / 58

The Human Eye / Summary Intermediate Summary Human Eye Similar components as a camera Photopic vision: Three types of photorecptors (cones) with different spectral characteristics Responsible for color vision Think about the following ... For representing color images, we use three different color components (e.g., RGB). How many color components would we need, if we wanted to design an image communication system for your ... dog (dichromatic vision) goldfish (tetrachromatic vision) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 13 / 58

Human Color Perception / What is Color ? The Nature of Light and Color Isaac Newton (1672): Color is a property of light, not of objects Sun light consists of 7 colored particles Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 14 / 58

Human Color Perception / What is Color ? Electromagnetic Waves Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 15 / 58

Human Color Perception / What is Color ? What Is Color ? what we see what our dogs/cats might see Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 16 / 58

Human Color Perception / What is Color ? Color: Interaction of Electromagnetic Waves with Human Cones  l( λ )  l( λ ) ⋅ Θ ( λ ) ∞ � ¯ L = l ( λ ) Φ( λ ) d λ L cones 0  m( λ )  m( λ ) ⋅ Θ ( λ ) ∞ � M = m ( λ ) Φ( λ ) d λ ¯ M cones 0 Θ ( λ )  s( λ ) ⋅ Θ ( λ ) ∞ �  s( λ ) S = s ( λ ) Φ( λ ) d λ ¯ 0 S cones Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 17 / 58

Human Color Perception / What is Color ? Human Color Perception What is Color? A certain spectral composition induces the perception of a particular color Color is not a physical quantity Color is a sensation in the viewer’s mind (interaction of electromagnetic waves with cones) Response of Human Cones to Electromagnetic Waves Electromagnetic spectra are mapped to a 3D vector ¯     L ∞ ℓ ( λ ) �  Φ( λ )  = M m ( λ ) ¯ d λ   Φ 0 S s ( λ ) ¯ 0 Φ( λ ) – Observed radiance spectrum Φ 0 – Arbitrarily chosen reference radiance ( Φ 0 > 0) ¯ l ( λ ) , ¯ m ( λ ) , ¯ s ( λ ) – Cone fundamentals (normalized spectral sensitivities) Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 18 / 58

Human Color Perception / Color Reproduction Metamers Φ 3 ( λ ) spectral radiance Φ ( λ ) Φ 1 ( λ ) perceived color: Φ 4 ( λ ) Φ 2 ( λ ) orange 400 450 500 550 600 650 700 750 wavelength λ [nm] Cannot distinguish light stimuli that yield the same cone excitation response ( L , M , S ) Light stimuli with that property are called metamers Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 19 / 58