High-Fidelity Light Field VR Playback Using NVIDIA GPUs Tim Milliron, Vice President of Engineering Nikhil Karnad, Architect for Image-Based Rendering 1
An Introduction to Light Field 2
THE LIGHT FIELD HOW IT WORKS KEY BENEFITS Capture the natural flow of every ray of Every pixel contains color, brightness and ⨠ ⨠ light depth properties (RGBZ) Leverages either hundreds of individual Every scene becomes a 3D model vs. a ⨠ ⨠ cameras or millions of microscopic lenses flat 2D image Requires tightly coupled hardware and Each captured perspective encodes ⨠ ⨠ software proper view-dependent illumination CONFIDENTIAL 3
Light Field for VR 4
THE LIGHT FIELD VOLUME ⨠ Capture ray data from every angle at all locations entering a given volume at high frame rate ⨠ Generate virtual views from any point within the volume, facing any direction, with any field of view. ⨠ Breakthrough sense of presence & realism for live action VR 5
THE LIGHT FIELD FOR VR - 6DOF ⨠ Parallax – Ability to “see behind objects” ⨠ View Dependent Illumination – Specular highlights, reflections, … ⨠ Truly Correct Stereo – Any viewing orientation - even when the viewer’s head is sideways – Any inter-ocular distance - adjustable on-the-fly 6
Lytro Immerge 7
LYTRO IMMERGE Rendering Capture Processing Playback • Rendering process to • Configurable multi- • Light-field playback in • Color matching generate final assets for camera system • Depth estimation leading HMDs efficient playback and compression
PRODUCTION-READY ⨠ Planar configuration comprised of 95 individual cameras ⨠ Nodal capture in 5 “wedges” ⨠ 475 cameras used to synthesize a full 360 view ⨠ Generates a 1-meter-wide “Viewing Volume” ⨠ Designed for modern high-end production ⨠ Director & film-crew can be behind the camera ⨠ Works well in practical on-set conditions ⨠ Highest resolution camera on the market today - up to 8k 360 resolution CONFIDENTIAL 9
CREATING A 360 LIGHT FIELD VOLUME 1 5 2 4 3 Lytro Immerge Light Field Merging Rotate to Capture “Wedges” Planar Configuration reconstructs the data into a five rotations to film a captures the environment from one 360 Light Field volume, enabling full 360 view of the environment direction creating a “wedge” 6DoF movement for viewers during playback CONFIDENTIAL 10
What’s the catch? 11
What’s the catch? 12
The Bad News 100-1,000x data 13
The Good News 14
The Good News 15
Light Field Capture & Playback is (barely) within reach today… 16
Lytro Immerge Rendering & Playback with NVIDIA GPUs 17
Light Field rendering: Requirements ⨠ Need two views, one per eye ⨠ High quality throughout the 6- DoF viewing volume ⨠ Close objects need to shift relative to far ones ⨠ Fill in occlusions seamlessly ⨠ Illumination variation across views should not be lost ⨠ Low latency, typically 90+ fps 18
Light Field rendering: Requirements ⨠ Need two views, one per eye ⨠ High quality throughout the 6- DoF viewing volume ⨠ Close objects need to shift relative to far ones ⨠ Fill in occlusions seamlessly ⨠ Illumination variation across views should not be lost ⨠ Low latency, typically 90+ fps Viewing volume DoF: Degrees of freedom 19
Light Field rendering: Requirements ⨠ Need two views, one per eye ⨠ High quality throughout the 6- DoF viewing volume ⨠ Close objects need to shift relative to far ones ⨠ Fill in occlusions seamlessly ⨠ Illumination variation across views should not be lost ⨠ Low latency, typically 90+ fps Viewing volume DoF: Degrees of freedom 20
Light Field rendering: Requirements ⨠ Need two views, one per eye ⨠ High quality throughout the 6- DoF viewing volume ⨠ Close objects need to shift relative to far ones ⨠ Fill in occlusions seamlessly ⨠ Illumination variation across views should not be lost ⨠ Low latency, typically 90+ fps Viewing volume DoF: Degrees of freedom 21
Light Field rendering: Requirements ⨠ Need two views, one per eye ⨠ High quality throughout the 6- DoF viewing volume ⨠ Close objects need to shift relative to far ones ⨠ Fill in occlusions seamlessly ⨠ Illumination variation across views should not be lost ⨠ Low latency, typically 90+ fps Viewing volume DoF: Degrees of freedom 22
Light Field rendering: Requirements ⨠ Need two views, one per eye ⨠ High quality throughout the 6- DoF viewing volume ⨠ Close objects need to shift relative to far ones ⨠ Fill in occlusions seamlessly ⨠ Illumination variation across views should not be lost ⨠ Low latency, typically 90+ fps Viewing volume DoF: Degrees of freedom 23
Light Field rendering: Background ⨠ Each wedge is a planar camera array ⨠ Captured rays sample the plenoptic function – Light slab [Levoy and Hanrahan, 1 1996] 5 2 – Lumigraph [Gortler et al., 1996] – Digital Light Field Photography 4 3 [Ng, 2006] – Light Field Camera Design [Wei et al., 2015] Theoretical two-plane Light Field parametrization 24
Light Field rendering: Background ⨠ Each wedge is a planar camera array Lytro Light Field ⨠ Captured rays sample the plenoptic camera function 2012 – Light slab [Levoy and Hanrahan, 1996] – Lumigraph [Gortler et al., 1996] – Digital Light Field Photography Lytro ILLUM [Ng, 2006] 2014 – Light Field Camera Design [Wei et al., 2015] 25
Lytro Immerge rendering: Practice ⨠ LOTS of captured rays! – Hundreds of cameras – Video capture rate 1 – Tens of billions of rays per 5 2 sec 4 3 ⨠ Per-ray payload multiplies data size Subset of the Light Field captured for Hallelujah 26
Lytro Immerge rendering: Practice ⨠ We use NVIDIA GPUs to crunch these massive datasets 1 2 5 ⨠ Assuming we could sift through 3 4 lots of rays per second during HMD render… ⨠ Upload to GPU would still be a bottleneck – Bandwidth requirement of hundreds of GB/s Image courtesy NVIDIA http://images.nvidia.com/pascal/img/titanx/titanx-design.png HMD: Head-mounted display 27
Lytro Immerge rendering: Acceleration ⨠ Goal – Unburden the GPU from having to consider the entire dataset ⨠ Solution – A proprietary acceleration structure that caches rays Eye center Camera center of perspective (in HMD) of perspective HMD: Head-mounted display 28
Lytro Immerge playback: Example 1 2 Limited field of view capture Left-eye and right-eye across two wedges views rendered to the HMD HMD: Head-mounted display 29
Lytro Immerge playback: Pipeline ⨠ Load: Cache allows order-of-magnitude improvement – Under 10 GB/s ⨠ Decode: Proprietary compression technique for further bandwidth reduction – Both CPU and GPU used ⨠ Shaders: Compute and graphics ⨠ OpenGL and DirectX implementations 30
Lytro Immerge playback: Performance ⨠ Require 90 fps or higher, i.e., 11 ms or faster ⨠ Careful balance between compute and graphics ⨠ Average GPU render times for Hallelujah – 980 Ti : 10.3 ms – Titan X Maxwell : 9.2 ms – Titan X Pascal : 6.5 ms 31
Acknowledgements ⨠ Representing the work of the entire LYTRO IMMERGE team ⨠ Shoutouts specifically for this section of the talk – Kurt Akeley, Trevor Carothers, Zeyar Htet, Derek Pang, Mike Ma, Alex Song, Cathy Ashenbremer ⨠ Contact information – Nikhil Karnad <nkarnad@lytro.com> – Tim Milliron @timmilliron 32
QUESTIONS & ANSWERS CONFIDENTIAL CONFIDENTIAL 33
High-Fidelity Light Field VR Playback Using NVIDIA GPUs Tim Milliron, Vice President of Engineering Nikhil Karnad, Architect for Image-Based Rendering 34
Recommend
More recommend
