Viruual Content Generation via Deep Learning Yinda Zhang Research - PowerPoint PPT Presentation

Viruual Content Generation via Deep Learning Yinda Zhang Research Scientist @ Google yindaz@google.com, www.zhangyinda.com GAMES Webinar

A bit about me...

A bit about me... 3D Scene Understanding Data-Driven Approach Shape Analysis

Depth Sensing on Device Image Depth Image Depth Pixel4 Rear Facing Camera: htups://ai.googleblog.com/2019/12/improvements-to-porurait-mode-on-google.html Pixel4 Front Facing Camera: htups://ai.googleblog.com/2020/04/udepth-real-time-3d-depth-sensing-on.html Zhang et.al., Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels, arXiv:2003.14299

Augmented Reality

A pipeline to generate viruual content... Generate 3D Manipulate 3D Render 3D Guo et.al., The Relightables: Volumetric Pergormance Wang et.al., Neural Pose Transfer by Spatially Adaptive Saito et.al., PIFu: Pixel-Aligned Implicit Function for Capture of Humans with Realistic Relighting, Instance Normalization, CVPR 2020. High-Resolution Clothed Human Digitization, ICCV 2019. SIGGRAPH 2019.

3D Shape Generation from a Single Image Input Volumes [1, 3] Point Cloud [2, 4] Mesh (Ours) Mesh + Texture (Ours) [1] Choy et.al., 3dr2n2: A unifjed approach for single and multi-view 3d object reconstruction, ECCV 2016. [2] Fan et.al., A point set generation network for 3d object reconstruction from a single image, CVPR 2017. [3] Lorensen et.al., Marching cubes: A high resolution 3d surgace construction algorithm, SIGGRAPH 1987. [4] Bernardini et.al., The ball-pivoting algorithm for surgace reconstruction, IEEE Trans. Vis. Comput. Graph 1999.

Pixel2Mesh

Input Volumes [1, 3] Point Cloud [2, 4] Mesh [5] Mesh [6] Implicit [7] Mesh (Ours) GT [1] Choy et.al., 3dr2n2: A unifjed approach for single and multi-view 3d object reconstruction, ECCV 2016. [5] Kato et.al., Neural 3d mesh renderer, CVPR 2018. [2] Fan et.al., A point set generation network for 3d object reconstruction from a single image, CVPR 2017. [6] Groueix et.al., Atlasnet: A papier-maˆche´ approach to learning 3d surgace generation, CVPR 2018. [3] Lorensen et.al., Marching cubes: A high resolution 3d surgace construction algorithm, SIGGRAPH 1987. [7] Mescheder et.al., Occupancy networks: Learning 3d reconstruction in function space, CVPR 2019. [4] Bernardini et.al., The ball-pivoting algorithm for surgace reconstruction, IEEE Trans. Vis. Comput. Graph 1999.

Pixel2Mesh

htups://walsvid.github.io/Pixel2MeshPlusPlus/

Shape Manipulation -- Pose Transfer

htups://jiashunwang.github.io/Neural-Pose-Transfer/

Neural Rendering Rely on texture map. Rely on volume. For Large Scale Scene Rely on implicit representation. Thies et.al., Deferred Neural Rendering: Image Synthesis using Neural Textures, SIGGRAPH 2019. Lombardi et.al., Neural Volumes: Learning Dynamic Renderable Volumes from Images, SIGGRAPH 2019. Mildenhall et.al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, arXiv:2003.08934

Neural Rendering Rely on texture map. Rely on volume. Rely on implicit representation. Rely on point cloud. Thies et.al., Deferred Neural Rendering: Image Synthesis using Neural Textures, SIGGRAPH 2019. Lombardi et.al., Neural Volumes: Learning Dynamic Renderable Volumes from Images, SIGGRAPH 2019. Mildenhall et.al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, arXiv:2003.08934

Neural Point Cloud Rendering RGB-D Scans Point Cloud Rendering

htups://daipengwa.github.io/NeuralPointCloudRendering_ProjectPage/

Neural Rendering Rely on texture map. Rely on volume. Rely on implicit representation. Rely on point cloud. Thies et.al., Deferred Neural Rendering: Image Synthesis using Neural Textures, SIGGRAPH 2019. Lombardi et.al., Neural Volumes: Learning Dynamic Renderable Volumes from Images, SIGGRAPH 2019. Mildenhall et.al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, arXiv:2003.08934

Implicit Function for 3D Park et.al., DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, CVPR 2019. Mescheder et.al., Occupancy Networks: Learning 3D Reconstruction in Function Space, CVPR 2019.

Render Implicit Function Haru et.al., Sphere tracing: A geometric method for the antialiased ray tracing of implicit surgaces. The Visual Computer 1996.

Render Deep Implicit Function ● Extremely time consuming ○ Too many queries for the rendering process. ○ Unroll multiple times for backpropagation. ○ Differentiable. Coarse-to-fjne Aggressive Marching Converge Criteria

Render Deep Implicit Function

htup://b1ueber2y.me/projects/DIST-Renderer/

Thanks!

Viruual Content Generation via Deep Learning Yinda Zhang Research - PowerPoint PPT Presentation

Viruual Content Generation via Deep Learning Yinda Zhang Research Scientist @ Google yindaz@google.com, www.zhangyinda.com GAMES Webinar A bit about me... A bit about me... 3D Scene Understanding Data-Driven Approach Shape Analysis Depth

Physical Attacks on Deep Learning Systems Ivan Evtimov Collaborators and slide content

Deep Learning Techniques for Music Generation 3. Generation by Feedforward Architectures

Deep Learning Techniques for Music Generation 3. Generation by Feedforward Architectures

Data Generation and Evaluation Using Deep Learning Ye Ji An a , Ji Hun Park a , So Hun Yun a , Man

DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs

Deep-Learning Oriented Smart Sensing for the Next Generation of Embedded Applications Manuele

TASO: Optimizing Deep Learning with Automatic Generation of Graph Substitutions Zhihao Jia , Oded

Automating High-Content Screening Image Analysis with Deep Learning GPU Technology Conference

Next Generation NAS Evolution of DSM Platform Jeremie Francois Synology Deep Machine Learning

Shallow Reading with Deep Learning Predicting popularity of online content using only its title W.

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

Layout Hotspot Detection with Feature Tensor Generation and Deep Biased Learning Haoyu Yang 1 ,

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning Techniques for Music Generation Recurrent (5) Jean-Pierre Briot

Deep Learning Techniques for Music Generation Recurrent (5) Jean-Pierre Briot

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Deep Learning Techniques for Music Generation Reinforcement (7) Jean-Pierre Briot

Deep Learning Techniques for Music Generation Compound and GAN (6) Jean-Pierre Briot

Deep Learning Techniques for Music Generation Compound and GAN (6) Jean-Pierre Briot

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Presentation title A Deep Learning based Fast Signed Distance Map Generation Zihao Wang, Clair

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin