TURING VR Davis Lin , - PowerPoint PPT Presentation

TURING 架构、实时光线追踪和 人工智能在 VR 中的技术应用 Davis Lin 林耀南 , Dec 19th

TURING 三大核心，通力合作 Tensor Core 核心处理器 Turing SM 核心处理器 每秒 114 兆次浮点运算 每秒 14 兆次浮点运算 + 每秒 14 兆次整数运算 每秒 228 兆次八位元整数精度运算 可同时执行浮点运算和整数运算 每秒 455 兆次四位元整数精度运算 可变尺度着色器 RT Core 光线追踪核心处理器 每秒 100 亿道光线追踪 Ray Triangle Intersection 射线与三角形相交检测 BVH Traversal 加速光线追踪效能 2

TURING ｜ 全新的 VR 计算平台 POWERING A NEW ERA OF VR 实时光线追踪和新 SM 架构 实时 AI 和 TENSOR CORE 5G AND CLOUD XR 全新的 VR 渲染方式 3

实时光线追踪和新 SM 架构 4

TURING RAY TRACING WITH RT CORES HARDWARE ACCELERATION REPLACES SOFTWARE EMULATION Turing SM Shaders RT Core Box Intersection Evaluators Launch Ray Probe Fetch box Decode box Intersection test Sub-box or tris? Triangle Intersection Evaluators Ray/triangle intersection test Shading Return hit 5

NEW PERFORMANCE STANDARD 1 TURING FRAME RAY TRACING 光线追踪运算 INT32 SHADING 整数运算 DNN PROCESSING FP32 SHADING 浮点运算 深度神经网络运算 6

NEW PERFORMANCE STANDARD 1 TURING FRAME RAY TRACING 光线追踪运算 INT32 SHADING 整数运算 DNN PROCESSING FP32 SHADING 浮点运算 深度神经网络运算 Tensor Turing Turing Turing Cores SM SM SM RT Cores 14 FP32 TFLOPS 14 FP32 TFLOPS 114 FP16 TENSOR FLOPS // + 100 TFLOPS 14 INT32 TIPS (10 TFLOPS PER GIGA RAY) 7

BULLET ICON Body/bullet text WITH a bullet icon option 8

9 9

Turing Pascal 2.0 TURING 架构 每时脉执行指令数（倍数） 1.5 1.5X 每时脉执行指令数 1.0 Shadow Rise of the 3DMark VR Deus Ex Sniper Elite Ashes of the 0.5 of War Tomb Mark 4 Singularity Raider Example Shader 10 10

11 11

实时 AI 和 TENSOR CORE 12

TENSOR CORE ACCELERATING COMPUTATION OF MATRIX MULTIPLES 13

TENSOR CORE ACCELERATING COMPUTATION OF MATRIX MULTIPLES 14

Up to 32X 15

PHOTO REALISTIC 3D RECONSTRUCTION Powered by Galasports InnoReal Photo-Realistic 3D Reconstruction system is a total solution for high precision 3D capture and rendering , it contains 5 modules: • Robotic Arm & Rail system for mobilize camera at 6 degree of freedom, cover subject in every angle. • High fidelity RGB & Depth camera system, capture RBG in 6k resolution ,and depth precision in 2 mm. 3D reconstruction and surface BRDF reconstruction • software system based on CNN & GAN Neural Network • Texture and BRDF compression software based on Neural Network • Realtime Rendering & streaming SDK compatible for iOS, Android, Windows, Linux and HTML5 16

InnoReal system not only reconstruct accurate geometry, but also reconstruct accurate surface reflection parameters according to PBR BRDF model 17

Diffuse Surface BRDF Neural Network Reconstructed Surface BRDF 18

19 InnoReal Trained a GAN model through 40k synthesized materials, the GAN network could predict BRDF parameters at each surface pixel according to input reflection samples of that surface pixel at multiple angle. The BRDF parameters are stored in a latent texture space, which could dramatically decrease texture data size. The frontend renderer decompress texture through a trained autoencoder and then render the texture and geometry by OpenGL 19

20 20

全新的 VR 渲染方式 21

VARIABLE RATE SHADING 可变码率渲染 Lens Optimized Shading Content Adaptive Shading Foveated Rendering VRS Wrapper APIs (VRS Helper and Gaze Handler) are available as a part of NVIDIA VRWorks Graphics 3.3 SDK. With the launch of the VIVE Pro Eye, HTC has created a Unity plugin, which has the VRS Wrapper NVAPIs integrated. 22

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 23

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 24

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 25

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 26

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 27

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 28

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 29

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 30

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 31

NVIDIA CONFIDENTIAL Use This for Slides That Include “Confidential” Information/Data NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 32

VRS WRAPPER UNITY PLUGIN HTC has developed a Plugin for Unity Once the native plugin is compiled as a DLL, it needs to be imported into Unity • Drag and drop the file from explore to Assets>Plugins filter/folder Confirm file is imported properly by noting path • To use the plugin we need to use C++ script that simply imports • functions from the DLL and calls the functions [DllImport("RenderingPlugin")] private static extern uint InitializeFoveatedRendering(); Note: In this example, “RenderingPlugin.dll” is the VRS Wrapper plugin DLL name 33

全球首款使用 VRS 技 术 的 FPS 游 戏 AEON 玩家将扮演第一位被捕捉的活跃生命意识信号，化身时间之神 - 伊 恩， 在虚拟世界里复活并逐渐找回能力，最终登录《奥丁号》的故事。 眼动追踪 + VRS + FPS = 新一代的游戏体验 34

7INVENSUN EYE TRACKING ACCESSORY Droolon F1 适配 Cosmos 、 Focus Plus 、 Focus 、 Vive Adapt to Cosmos,Focus Plus,Focus,Vive VR 社交 VR Social 心理康复 Psychological Rehabilitation 眼动交互 Eye Movement Interaction 眼动数据分析 视觉训练 注视点渲染 Eye Movement Data Analysis Visual Training Foveated Rendering 35

“烛龙杯” 眼球追踪 VR 内容开发大赛 本次大赛是围绕 HTC VR 头显（含 Cosmos 、 Focus Plus 、 Vive Pro 等） 和七鑫易维眼球追踪产品举办的虚拟现实（ VR ）眼球追踪内容开发大 赛，以下简称“大赛”。 主要目的是充分挖掘眼球追踪技术在 VR 中的价值，在 VR 内容中利用眼 球追踪技术，如注视点渲染、眼控交互、眼动数据分析、 Avatar 眼部 复刻、心理康复、视觉训练等，但不限于以上内容。 参赛作品须充分挖掘眼球追踪技术在 VR 中的价值，体现实用性和创新 性，为 VR 内容带来更好的体验，引领 VR 内容新方向的探索。 赛事启动 报名阶段 作品提交 专家评审 颁奖活动 报名之日起 — 2020 年 4 月 15 日 2020 年 4 月 16 日 — 2020 年 5 月 15 日 2020 年 5 月中下旬 2019.12.18 2019.12.18 — 2020.2.15 36

TURING: MULTI-VIEW RENDERING 支持四个不同角度的 Views 满足更大的视场角 Left Eye Right Eye 37

VR SLI Multi-GPU Accelerated Immersive Rendering Left view data L R Geometry Materials Right view data R 38

BYKER VR ANATOMY IN MEDICAL EDUCATION Multi-GPU Accelerated Immersive Rendering 39

NVIDIA VRWorks Comprehensive SDK for VR Developers GRAPHICS HEADSET PROFESSIONAL SIMULATION VIDEO 40 40

41 41

5G AND CLOUD XR 42

THE OPPORTUNITY TO PROVIDE VR/AR/VGPU OVER 5G A Convergence of Visual Compute and 5G and a new Edge Topology Powerful GPUs Bandwidth and Low Latency Compute is at the Edge Incredible compute capabilities redefining Both are required for VR/AR Very few hops – Low Latency graphics 43

ENTERPRISE MARKET PENETRATION The Debate of Whether VR is Viable in Enterprise is Over ESI Enscape Training AEC Possibly the Largest Entire Design Process, Market Sales, Moving to IC.IDO Nebulize Construction M&E Virtual Production Autodesk Automotive Location-Based Entertainment 44 Engineering, Design, Marketing/Sales Enterprise/Consumer

ENTERPRISE AR Driving Value Propositions INSTRUCTION/COLLABORATION DIGITAL MODELS ANNOTATED 45

5G + VR/AR/VGPU AT THE EDGE IS A SIGNIFICANT OPPORTUNITY Requirements: • Ultra Low Latency • High Bandwidth • Cutting-edge Compute • Cost Effective Delivery • Valuable Use Cases 46