THE VIRTUAL FRONTIER: Computer graphics challenges in virtual - PowerPoint PPT Presentation

THE VIRTUAL FRONTIER: Computer graphics challenges in virtual reality Dr. Morgan McGuire, NVIDIA Research May 9, 2017 GPU Technology Conference

VISION 1977 HPC 2017 HPC 1997 HPC Today, everyone is a high-performance computer user, with GPUs in phones, tablets, desktops, game consoles, and cars 2

VISION Power User Technology Pervasive FUTURE VR 3

1. Virtual reality will be the new interface to computing for everyone 2. Virtual reality requires a new graphics system sensors, algorithms, data structures, processors, and displays 4

MODERN GRAPHICS SYSTEMS 5

VISUAL FIDELITY OF FILM CGI Deadpool (Marvel) 6

Deadpool (Marvel) 7

FILM CGI: CONCEPT TO PHOTONS Direction Performance Modeling Script Screen Rigging Particles Characters Composite & Path Tracer 24 Hz Animation Triangles Costumes Color Grade 9 Mpix Texturing Sub-D Sets Lighting (Hours) Simulation Preproduction Production Primitives Renderer Display 8

FILM CGI: CONCEPT TO PHOTONS Direction Performance Modeling Script Screen Rigging Particles Characters Composite & Path Tracer 24 Hz Animation Triangles Costumes Color Grade 9 Mpix Texturing Sub-D Sets Lighting (Hours) Simulation Preproduction Production Primitives Renderer Display 9

3D GAME SYSTEM Direction Primitives Renderer Preproduction Production Monitor Performance Modeling Script Rigging Characters Display AI Animation Shadow HDMI, Particles Costumes Rasterization AO Shade Post FX* 30Hz Network Texturing Maps Sync Triangles Sets Lighting 2 MPix Simulation Simulation User Input 10

3D GAME SYSTEM Direction Primitives Renderer Preproduction Production Monitor Performance Modeling Script Rigging Characters Display AI Animation Shadow HDMI, Particles Costumes Rasterization AO Shade Post FX* 30Hz Network Texturing Maps Sync Triangles Sets Lighting 2 MPix Simulation Simulation User Input 11

3D GAME SYSTEM Primitives Renderer Monitor Display AI HDMI, Shadow Particles Rasterization AO Shade Post FX* 30Hz Network Sync Maps Triangles 2 MPix Simulation User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 12

3D GAME SYSTEM Primitives Renderer Monitor Display AI HDMI, Shadow Particles Rasterization AO Shade Post FX* 30Hz Network Sync Maps Triangles 2 MPix Simulation User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 13

14 Star Wars: Battlefront II (DICE)

15 Forza Motorsport 6 (Turn 10 Studios) 15

7X THROUGHPUT INCREASE 3D GAME = 60 MPIX/S MODERN VR = 450 MPIX/S ( 3024 X 1680* @ MIN 90 FPS) ( 1920 X 1080 @ MIN 30 FPS) 1680 1080 1920 1512 1512 16 *VR render resolution

Primitives Renderer Monitor Display AI HDMI, Shadow Particles Rasterization AO Shade Post FX* 90Hz Network Sync Maps Triangles 5 MPix Simulation User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 17

MODERN VR SYSTEM Primitives Renderer HMD Head Tracking Display AI HDMI, Particles Shadow Time Warp Rasterization 90Hz Network Sync Triangles Maps 5 MPix Simulation User Input and Tracking 18

LENS DISTORTION Rendered Image Optics User’s View 19

MODERN VR SYSTEM Primitives Renderer HMD Head Tracking AI HDMI, Time Warp + Particles Shadow Display Rasterization Network Sync Lens Distortion Triangles Maps 90Hz Simulation User Input and Tracking 20

1. Virtual reality will be the new interface to computing for everyone 2. Virtual reality requires a new graphics system sensors, algorithms, representations, processors, displays 3. Pascal architecture upgrades the gaming system to modern VR warping, lens matched shading, multiprojection, stereo projection, variable resolution 21

FUTURE GRAPHICS SYSTEMS The remainder of the talk describes active research, including new results not previously presented in public. These are not products. 22 22

LIMITS OF HUMAN PERCEPTION 100,000x to 1Mx Beyond Modern VR 220 o Horizontal x 135 o Vertical x (120 pixels/degree) 2 ≈ 400,000,000 pixels = 200 x 1080p TVs x 240 fps Future VR = 100,000 Mpix/s Modern VR = 450 Mpix/s + High dynamic range (x2), photorealistic dynamic lighting (x10,000), … 23 Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html

FOVEATED RENDERING Primitives Renderer HMD Head Tracking AI HDMI, Time Warp + Particles Shadow Foveated Display Network Sync Lens Distortion Triangles Maps Rasterization 90Hz Simulation Eye Tracking User Input and Tracking 24

FOVEATED RENDERING Conventional Approach: Aliasing Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016 25

FOVEATED RENDERING Our Approach: Perceptually Optimized Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016 26

BEYOND TRIANGLES Primitives Renderer HMD Particles Head Tracking Triangles Points AI HDMI, Time Warp + Shadow Foveated Display Text Network Sync Lens Distortion Maps Rasterization 90Hz Voxels Simulation Light Fields Eye Tracking User Input and Tracking 27

BEYOND TRIANGLES Light Fields 28 McGuire et al., Real-time global illumination with light field probes, I3D 2017

COMPUTATIONAL DISPLAYS Primitives Renderer HMD Particles Head Tracking Triangles Points AI HDMI, Varifocal Shadow Foveated Display Text Time Warp Network Sync Lens Distortion Maps Rasterization 90Hz Voxels Simulation Light Fields Light Field Eye Tracking User Input and Tracking 29

COMPUTATIONAL DISPLAYS Light Field Display Display Prototype GPU Output Observed Image Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013 30

COMPUTATIONAL DISPLAYS Varifocal Optics Hologram 31 Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies

COMPUTATIONAL DISPLAYS Varifocal Optics 32 Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017

PNEUMATIC HAPTICS Primitives Renderer HMD Light Field Particles Head Tracking Triangles Points AI Text HDMI, Varifocal Shadow Foveated Display Time Warp Network 2 ½ D Video Sync Lens Distortion Maps Rasterization 90Hz Simulation Voxels Light Fields Eye Tracking User Input and Tracking Haptics Shepherd et al., Stretchable Transducers for Haptic Interactions in Virtual Reality, 33 GTC VR Village 2017 & SIGGRAPH Emerging Technologies, 2017

LOW LATENCY Hierarchical Rendering Primitives Renderer HMD Remote Particles Light Field GPU Head Tracking Triangles Points AI Text HDMI, Varifocal Shadow Foveated Display Time Warp Network 2 ½ D Video Sync Lens Distortion Maps Rasterization 90Hz Simulation Voxels Light Fields Eye Tracking User Input and Tracking Haptics 34

LOW LATENCY Hierarchical Rendering Near-HMD Tegra Local GeForce Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015 35

LOW LATENCY Hierarchical Rendering Cloud GRID Platform Near-HMD Tegra Local GeForce Tesla GPU Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015 36

LOW LATENCY Hierarchical Rendering High speed network Compressed lighting data Cloud GRID Platform Near-HMD Tegra Tesla GPU Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015 37

LOW LATENCY Binary Frames Primitives Renderer HMD Remote Head Tracking Particles GPU Triangles Light Field Points Time Warp + Deep Focus AI Text Shadow Foveated HDMI, Varifocal Display Network 2 ½ D Video Maps Rasterization Sync Lens Distortion 16000Hz Simulation Voxels Light Fields Eye Tracking User Input and Tracking Haptics 38

ULTRA LOW LATENCY & HIGH THROUGHPUT Binary Frames Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016 39

LOW LATENCY Binary Frames Binary Frames 0.08 ms Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016 40 Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016

LOW LATENCY On-HMD Warping Hardware Warp Prototype Warped Static Point Set Head-Mounted Display 41

RAY & PATH TRACING Primitives Renderer HMD Remote Head Tracking Particles GPU Triangles Light Field Points Time Warp + Deep Focus AI Shadow Foveated HDMI, Text Denoising Varifocal Display Network Maps Path Tracing Sync Voxels Lens Distortion 16000Hz Simulation Light Fields Eye Tracking User Input and Tracking Haptics 42

PATH TRACING 10 rays/path Deadpool (Marvel) 43

PATH TRACING 10 rays/path 2 paths/pixel Deadpool (Marvel) 44 Visualization of path tracing noise

10 rays/path 1000 paths/pixel 45