Early visual processing for CG Early visual processing for CG Rich Clarke
Imagine we are building VR system. Money is no object: • We are asked to specify the ultimate, fully immersive visual display device. We are asked to specify the ultimate, fully immersive visual display device. • What is the minimum that they must do, so that the resulting Virtual World is utterly indistinguishable from reality? • I.e. what are the hard limits on our perception of reality?
How to start to answer this question? • Basic visual hardware: Visual anatomy & physiology Basic visual hardware: Visual anatomy & physiology • The fundamental unit of visual coding: Receptive fields • Visual pathways: The split of information streams p y p • Brief aside: How do photoreceptors change EM radiation into neural activity? • How well can you really see? Resolution and acuity • Seeing the sun and the stars: Brightness adaption • Visual masking • Colour perception We’re going to keep in mind: The So What? Implications for Computer Graphics? Graphics?
Basic visual hardware: Visual anatomy & physiology The eye… [Ferwerder] [ ] ANTERIOR: Modelled as camera optics… POSTERIOR: ~Hexagonal mesh of POSTERIOR: ~Hexagonal mesh of photoreceptors…
Basic visual hardware: Visual anatomy & physiology Photoreceptors in the retina: Rods & Cones Pigmented layer Start of the ‘brain’… To the optic nerve LIGHT LIGHT Adapted from: [Ferwerder]
Basic visual hardware: Visual anatomy & physiology Photoreceptors in the retina: Rods & Cones (2) • Extremely sensitive to light • Provide achromatic vision Provide achromatic vision • Work at low level (scotopic) illumination • Large receptive fields g p • Peak absorbance (sensitivity) at ~500nm • Less sensitive to light Less sensitive to light • Provide colour vision • Work at high level (photopic) illumination • Three types: ‘B’ peak at 437nm, ‘G’ peak at 533nm, ‘R’ peak: 564nm Much smaller receptive fields [Ferwerder]
Basic visual hardware: Visual anatomy & physiology SO WHAT? • Understanding of “hardware” gives insight into kinds of information that U d t di f “h d ” i i i ht i t ki d f i f ti th t can be coded • Real VR system: focus resource on right areas y g
The fundamental unit of visual coding: Receptive fields A Cell s Receptive Field A Cell’s Receptive Field ‘On centre’ cells • Defined by spatially localised group of photoreceptors serving some ganglion cell photoreceptors serving some ganglion cell Higher up in ‘Off the brain (in centre’ • Location and quality of stimulus to which V1): integrate cells cells simple cells: simple cells: the ganglion cell is responsive the ganglion cell is responsive complex cells • Opponency: On centre and off centre On centre and off centre Spectral as well as spatial: Red/Green Yellow Blue Eye Absolute physical values lost: Absolute physical values lost: [Ferwerder]
The fundamental unit of visual coding: Receptive Fields SO WHAT? • Basic building blocks of visual perception B i b ildi bl k f i l ti • Information about absolutes (both brightness and ‘colour’) lost – contrast & context sensitivity only (c.f. illusions) y y ( ) • Brain ‘looks for’ fundamental structures that are/have been behaviourally relevant in ontogenetic and phylogenetic history: fast ‘h ‘hardware’ recognition d ’ iti
Visual pathways: The split of information streams From the eye to the brain [Ferwerder] [Ferwerder] LGN 6 l LGN: 6 layers: Magnocellular layers – primary input from peripheral retina – non spectrally opponent ganglions large receptive fields non spectrally opponent ganglions, large receptive fields Parvocellular layers – primary input from the foveal region – spectrally opponent cells small receptive fields spectrally opponent cells, small receptive fields Area V1 (Visual Cortex): more complex cells (I.e. with more complex receptive fields) Deals with What? And Where? …Separately?
Visual pathways: The split of information streams SO WHAT? Eyes may be serving 2 relatively separate visual systems: Eyes may be serving 2 relatively separate visual systems: • Fast response, achromatic system, motion sensitive, low res. (Magnocellular layers) (Magnocellular layers) • Slow response trichromatic system, motion insensitive, high res. (Parvocellular layers)
How do photoreceptors change EM radiation into neural activity? Rods & Cones: Outersegment: billions of light sensitive pigment molecules Outersegment: billions of light sensitive pigment molecules Molecules embedded in disks, stacked like pancakes Rods: Pigment is Rhodopsin γ
How well can you really see? Resolution and acuity Physical limits on resolving power Three things determine resolution: 1. Optical filtering, 2. receptor sampling, (and 3. receptive field organisation) 1. Real optical system: aberrations, diffraction at entry aperture Resolution limited to 30arcsec 2. Photoreceptors sample retinal image -> neural image representation. Spacing well matched to optics (Sampling theory) Visual acuity is a function of contrast sensitivity ~30sec Vernier (hyper)acuity: Ability to localise position of objects – not a function of contrast sensitivity Can detect misalignments of ~5sec Unknown exactly how it’s done [Ferwerder]
How well can you really see? Resolution and acuity SO WHAT? • Obviously sets hard limit on how much detail required of a Obviously sets hard limit on how much detail required of a display system • VR systems not close to this for real-time display • Vernier acuity plays an important role in the visibility of aliasing artefacts in digital images – simple analysis of the visual system would predict that some artefacts should not be seen system would predict that some artefacts should not be seen (below the limit of supposed visual acuity)
Seeing the sun and the stars: Brightness adaption Brightness adaption g p How many orders of magnitude difference between the dimmest and the How many orders of magnitude difference between the dimmest and the brightest things we can see? [Ferwerder]
Seeing the sun and the stars: Brightness adaption Brightness adaption (2) g p ( ) Three mechanisms Mechanical --- Pupil dilation Photochemical --- Bleaching & regeneration Neural Neural --- --- Changes in processing Changes in processing Changes in: Contrast Sensitivity: Ch i C t t S iti it P tt Pattern Acuity: A it C l Colour Perception P ti All: [Ferwerder]
Seeing the sun and the stars: Brightness adaption The time course of adaption p Purkeinje break [Ferwerder]
Seeing the sun and the stars: Brightness adaption SO WHAT? • Most of the information or 'power' (in Fourier domain) of an Most of the information or power (in Fourier domain) of an image is in brightness contrast • Using 3 adaption mechanisms, able to see effectively over a range of ~10 log units • At different luminances contrast sensitivity, acuity, colour perception changes markedly perception changes markedly • Obvious implications for the design of a VR system (resource allocation etc.)
Visual masking Masking… g Defn: Visual Masking: Presence of 1 visual pattern changes visibility of another Brightness: Top: 8 bit quantisation p q Bottom: 4 bit quantisation Experiments performed to determine parameters of visual masking Critical Band masking: Masking effect is greatest when frequency of the masking pattern is adjacent to the frequency of the i image spectrum t [Ferwerder]
Visual masking SO WHAT? • Good example of how we do not see the world in veridicial Good example of how we do not see the world in veridicial manner: masking changes way in which features perceived • Contrast nonlinearity – implications for way that features of world are coded by visual system
Colour perception [Purves & Lotto]
Colour perception [Purves & Lotto]
Colour perception What is colour perception? How do we (efficiently) recreate it? p p ( y) Relative stimulation of each cone type in your retina (RGB) in the context of some visual field visual field Different spectral distributions of light should be able to stimulate the photoreceptors identically: photoreceptors identically: Spectral distributions Receptor sensitivity SD a SD b SD b Distinct distributions that are perceived identically w.r.t some visual system - METAMERS
Colour Perception SO WHAT? • Sensible choice of some colour primaries should allow you to Sensible choice of some colour primaries should allow you to re-create any visible colour simply (without recreating the whole C( λ ) distribution • Not quite as simple as that…but right primaries will produce a colour gamut that covers most visible colours
Recommend
More recommend
