PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anj ul Patney S - PowerPoint PPT Presentation

PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anj ul Patney S enior Research S cientist

INTRODUCTION • Virtual reality is an exciting challenging workload for computer graphics • Most VR pixels are peripheral • Efficient peripheral rendering is poorly understood • Quick perceptual experiments can expose opportunities for significant speedups 2

FOVEATED RENDERING Foveal Pixels Peripheral Pixels 3

Why foveated rendering? THIS TALK S tate of the art in efficient peripheral rendering How else could we reduce peripheral details? 4

WHY FOVEATED RENDERING? 5

PC Gaming : 60 MP/s ( 1920 x 1080 @ 30 Hz) 1080 1920 6 *VR render resolut ion f or HTC Vive

Virtual Reality : 450 MP/s PC Gaming : 60 MP/s ( 3024 x 1680* @ ( 1920 x 1080 @ 30 Hz) 90 Hz) 1080 1680 1920 1512 1512 7 *VR render resolut ion f or HTC Vive

VR IS UP TO 7X MORE DEMANDING TODAY Virtual Reality : 450 MP/s PC Gaming : 60 MP/s ( 3024 x 1680* @ ( 1920 x 1080 @ 30 Hz) 90 Hz) 1080 1680 1920 1512 1512 8 *VR render resolut ion f or HTC Vive

MOST VR PIXELS ARE PERIPHERAL 4% Column1 20° 73% Foveal 96% Peripheral Peripheral iPhone 7 27" Deskt op 2016 VR Plus Monitor HMD 9

MOST VR PIXELS ARE PERIPHERAL 4% Column1 Efficient 20° peripheral 73% Foveal rendering can 96% unlock significant Peripheral Peripheral performance gains iPhone 7 27" Deskt op 2016 VR Plus Monitor HMD 10 10

EFFICIENT PERIPHERAL RENDERING What is the ideal peripheral image? How to efficiently render the ideal peripheral image? 11

STATE OF THE ART IN EFFICIENT PERIPHERAL RENDERING 12

MULTI-PASS FOVEATED RENDERING [Guenter et al. 2012] 13 13

GPU-ACCELERATED FOVEATED RENDERING NVIDIA Multi-Res S hading 14

GPU-ACCELERATED FOVEATED RENDERING NVIDIA Lens-Matched S hading 15

BEYOND FOVEATED IMAGE RESOLUTION • Peripheral vision shows non-uniform 100K Rods optical, retinal, and neural degradation Density (per mm2) 10K Cones • Over an order of magnitude potential improvement 1K 100 Ganglion Cells 10 0 20 40 60 80 Eccentricity (degrees) 16

BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High S patial Color Flicker Motion Frequencies Faces Crowded S timuli http://pics.psych.stir.ac.uk/ 17 17

BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High S patial Color Flicker Motion Frequencies Faces Crowded S timuli http://pics.psych.stir.ac.uk/ 18 18

BEYOND FOVEATED IMAGE RESOLUTION LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD High S patial Color Flicker Motion Frequencies Faces Crowded S timuli http://pics.psych.stir.ac.uk/ 19 19

PERCEPTUAL FOVEATED RENDERING [S tengel et al. 2016] Acuity Fall-off, Eye/ Obj ect Motion, Contrast, Brightness Adaptation 2-3x lower fragment shading workload 20 20

PERIPHERAL BLUR PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 21 21

PERIPHERAL BLUR + CONTRAST PRESERVATION PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 22 22

PRACTICAL RENDERING SYSTEM WITH 2X-3X FEWER PIXEL SHADES PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016] 23 23

HOW ELSE COULD WE REDUCE PERIPHERAL DETAILS? 24

PERCEPTUAL HYPOTHESES “ Can we reduce peripheral color saturation? ” “ Can we foveate alternate frames? ” “ S hould we foveate symmetrically around the gaze center? ” 25

EARLY EXPERIMENTS • S imulated foveation following each hypothesis • Performed ad-hoc Yes/ No tests with two subj ects • Estimated threshold using MATLAB psychtoolbox • Total time per trained subj ect: < 1 hour 26

BASELINE: FOVEATED BLUR 27

CAN WE REDUCE PERIPHERAL COLOR SATURATION? 28

CAN WE FOVEATE Even Frame Odd Frame ALTERNATE FRAMES? 29

CAN WE FOVEATE Even Frame ALTERNATE FRAMES? 30

SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER? Temporal Foveation Nasal Foveation 31 31

SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER? Temporal Foveation 32 32

RESULTS – MAGNITUDE OF FOVEATION 50 S ubj ect 1 S ubj ect 2 40 Foveation threshold 30 20 10 0 Baseline Color Flicker Nasal Temporal 33 33

RESULTS – ESTIMATED PERFORMANCE 1.6 1.4 Relative performance 1.2 1 0.8 0.6 0.4 0.2 0 Color Flicker Nasal + Temporal 34 34

RESULTS – ESTIMATED PERFORMANCE 1.6 Potential 1.4 Relative performance Win! 1.2 1 0.8 0.6 0.4 0.2 0 Color Flicker Nasal + Temporal 35 35

SUMMARY • Efficient peripheral rendering is fundamental for VR rendering • But we have a lot to learn about efficient peripheral rendering • What is the ideal peripheral image? • How to efficiently render the ideal peripheral image? • Perceptual evaluations can expose opportunities for significant speedups 36

ACKNOWLEDGMENTS Joohwan Kim Chris Wyman David Luebke Rachel Albert Marco S alvi Aaron Lefohn Peter S hirley Jason Paul 37

THANK YOU apatney@ nvidia.com 38