ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Cem Cebenoyan Edward Liu 1
ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Talk Overview • VRWorks Overview Graphics, audio, video stitching, physics Lens Matched Shading Deep Dive • Fundamental technology Engine integration details 2
NVIDIA VRWORKS Bringing Reality to VR SIGHT SOUND PHYSICS & TOUCH CAPTURE SIGHT SOUND TOUCH BEHAVIOR PHYSX VRWorks 360 Video Simultaneous Ray Traced PhysX & CUDA Multi-projection Audio 3
COMPUTING CHALLENGES IN REPRODUCING REALITY PHYSICS & GRAPHICS AUDIO CAPTURE TOUCH 4
VR RENDERING Ultra-High Resolution and Frame Rate TRADITIONAL = 60 MP/S VIRTUAL REALITY = 450 MP/S ( 1920 X 1080 @ 30 FPS) ( 3024 X 1680* @ 90 FPS) 1080 1680 1920 1512 1512 *VR render resolution 5
VR PERFORMANCE DEMANDS Ultra-Low Latency Motion to Photon: ≤ 20 ms 6
VR OPTICS Optics LCD display User’s view 7
VR RENDERING Rendered Image Warped Image 8
VR RENDERING GPU renders many pixels that never make it to the screen Warped Image Rendered Image 9
VRWORKS MULTI-RES SHADING 10
VRWORKS LENS MATCHED SHADING Renders to a lens corrected surface 11 11
VRWORKS MRS AND LMS Fast viewport broadcast hardware on NVIDIA Maxwell and beyond GPUs Viewport 1 Viewport 2 Geometry Pipeline ... Viewport N 12
TRADITIONAL STEREO RENDERING Requires 2 geometry passes 13 13
VRWORKS SINGLE PASS STEREO Renders left & right eye in one geometry pass Left Eye Right Eye 14 14
VRWORKS VR SLI Scales performance across multiple GPUs Frame 1 (Left eye) Warped Frame Frame 1 (Right eye) Head Tracking (t) Head Tracking (t+1) 15
“ Normal ” SLI GPUs render alternate frames CPU N N+1 GPU 0 N GPU 1 N+1 Display N N+1 Latency
VR SLI Each GPU renders one eye — lower latency N N+1 CPU N L N+1 L GPU 0 N R N+1 R GPU 1 Display N N+1 Latency
VR SLI GPU affinity masking: full control Left eye rendering Shadow maps, GPU physics, etc. Right eye rendering UINT SetGPUMask( void RenderGPUMaskNV( [in] UINT GPUMask [in] bitfield mask ); );
VR SLI Per-GPU constant buffers, viewports, scissors L L R Multi-GPU Engine API R void MulticastBufferSubDataNV( NvAPI_Status VSSetConstantBuffers( bitfield gpuMask, [in] ID3D11DeviceContext *pContext, uint buffer, [in] UINT GPUMask, intptr offset, [in] UINT StartSlot, sizeiptr size, [in] UINT NumBuffers, const void *data ); );
VRWORKS BRINGS MAJOR SPEEDUPS TO TOP VR APPS Without VRWorks With VRWorks 2.0 1.6 Relative Performance 1.2 0.8 0.4 0.0 Funhouse Everest Raw Data SportsBar VR Trials of Tatooine *Performance measured on GeForce GTX 1080 using VRWorks MRS, LMS, or VR SLI
COMPUTING CHALLENGES IN REPRODUCING REALITY GRAPHICS / PHYSICS & AUDIO CAPTURE DISPLAY TOUCH 21 21
SIMULATING AUDIO IN VR SYNTHESIS DIRECTION PROPAGATION Creation of Source Sounds Location of Incoming Sound How Sound Moves in Space 22 22
AUDIO REFLECTS YOUR ENVIRONMENT Elevator Symphony Hall Meadow 23 23
24 24
25 25
Models direction and propagation using Ray tracing 26 26
VRWORKS AUDIO SDK Features: NOW AVAILABLE! • Directionality and Reverbs • Sound Propagation with Attenuation, Reflection, Diffraction, Transmission • Realistic Occlusion • Dynamic Geometry Easy To Use Presets: • Effect Strength • Materials • Compute Available as SDK and Plugin for UE 4.15+ 27 27
COMPUTING CHALLENGES IN REPRODUCING REALITY GRAPHICS / PHYSICS & AUDIO CAPTURE DISPLAY TOUCH 28 28
HAPTICS Collision detection & deformation modeling FORCE FRICTION RESTITUTION PhysX API PhysX Constraint Solver Haptics Layer 29
REALISTIC PHYSICS Simulating behavior in VR PHYSX FLEX DESTRUCTION HAIRWORKS FLOW CLOTH 30 30
31 31
COMPUTING CHALLENGES IN REPRODUCING REALITY GRAPHICS / TOUCH / AUDIO CAPTURE DISPLAY PHYSICS 32 32
SIGNIFICANT COMPUTATION REQUIRED TO DELIVER 360 VIDEO Display Capture Stitch Decode Calibrate Equalize Stitch Encode 4k cameras Single 360 video 33 33
"The fact that NVIDIA manages to stitch 4K 360 VRWORKS 360 VIDEO SDK stereoscopic video in real time making live streaming possible changes the production pipeline NOW AVAILABLE! and enables entirely new use cases in VR“ Kinson Loo, CEO of Z CAM. Features - Real-Time & Offline Stitching - Up to 32 x 4k Camera Rigs GPU Accelerated Pipeline Decode -> Calibration -> Equalization -> Stitching -> Encode Mono SDK Available in Beta Now! Stereo SDK Available Soon 34
VRWORKS FOR UE4 AVAILABLE NOW Features: Multi-res Shading Lens Matched Shading Single Pass Stereo VR SLI Available for 4.12 through 4.15 Download at: developer.nvidia.com/nvidia- vrworks-and-ue4 35
VRWORKS FOR UNITY AVAILABLE NOW Features: Multi-res Shading Lens Matched Shading Single Pass Stereo VR SLI Supported in Unity 2017.1 Beta 2 Download VRWorks plugin at www.assetstore.unity3d.com 36
LENS MATCHED SHADING DEEP DIVE 37
38
39
40
41
SHADING RATE AND LENS WARP Two effects of applying lens warp to an image: 1.Periphery squashed 2.Central region magnified This distortion makes image center undersampled and periphery supersampled 42
SHADING RATE AND LENS WARP 43
SHADING RATE AND LENS WARP Therefore VR applications usually render at a higher resolution than display resolution. • HTC Vive Display Resolution 2160x1200, Render Resolution 3024x1680 • Oculus Rift Display Resolution 2160x1200, Render Resolution 2664x1586 44
SHADING RATE AND LENS WARP 45
LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 46
LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 47
LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 48
LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 49
LENS MATCHED SHADING BREAKDOWN 50
LENS MATCHED SHADING BREAKDOWN 51
PRE-SET CONFIGURATIONS We provide 3 sets of configurations for both HTC Vive and Oculus Rift 1.Quality: No undersampling accross the image (while reducing total # of pixels) 2.Conservative: Undersampling no worse than baseline 3.Aggressive: ¾ Resolution of the Conservative config (keeping center shading rate high) 52
PRE-SET CONFIGURATIONS We also provide a scalar variable that smoothly change the shading rate It keeps the center shading rate constant Allows finer grain tuning between image quality and performance 53
LMS VS. MRS The 1/x profile of LMS can closer approximate the lens profile than the piecewise constant of MRS LMS needs fewer shading to achieve the same image quality with MRS LMS has a smoother shading rate transition across the image LMS uses 4 viewports per eye, while MRS uses 9. This makes LMS easier to work along with Instanced Stereo and Single Pass Stereo Fewer viewports also benefits performance 54
LMS VS. MRS shading Desired (inverse lens warp) rate Default (planar projection) MRS (Quality) 1.35 LMS (Quality) 1.04 1.0 (screen) x 55
Conservative Aggressive Quality LMS vs. MRS (no undersampling) (no worse than baseline) (3/4 Reso. of conservative) LMS 1.17 MPix / eye 0.87 MPix / eye 1.57 MPix / eye Baseline MRS (no warp) 2.54 MPix / eye 1.58 MPix / eye 2.03 MPix / eye 1.40 MPix / eye 56
LMS/MRS Shading Rate Comparison 3 LMS MRS Baseline 2.54 Number of Pixels (Millions) 2.5 2.03 2 1.58 1.57 1.5 1.4 1.17 1 0.87 0.5 0 Baseline Quality Conservative Aggressive 57
ENGINE INTEGRATION OVERVIEW 1. Fill Gbuffer in LMS Space 2. Shading in LMS Space 3. Post Processing in LMS Space 4. Resample to Linear Space 58
LMS COORDINATES SYSTEM (0, 0) (1, 0) (0, 0) (1, 0) (0, 0) (0.5, 0) (0.5, 0) (1, 0) (0, 1) (0.5, 1) (0.5, 1) (1, 1) (0, 1) (1, 1) (0, 1) (1, 1) Linear Space UV LMS Space UV LMS Space Stereo UV Z value changed too! 59
Recommend
More recommend
