Neural Rendering Chuan Li Lambda Labs Collaborators: Thu - PowerPoint PPT Presentation

Neural Rendering Chuan Li Lambda Labs Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei Zhang, Rui Wang, Kun Xu, Rui Tang

Forward (Computer Graphics) Model Pictures

Forward (Computer Graphics) Model Pictures Inverse (Computer Vision)

Integral of the incident radians

BRDF

32K SPP Ray Tracing (90 mins 12 CPU Cores) The Tungsten Renderer

P 0 P 1

P 0 P 1

P 0 P 1 R 01 | T 01

Inverse (Computer Vision) P 0 P 1 R 01 | T 01

Inverse (Computer Vision) P 0 P 2 P 1 R 01 | T 01 R 12 | T 12

Building Rome in a Day Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski

Sub-module End-2-End Model Pictures Differentiable Rendering

1 SPP 2048 SPP

Sub-modules Mastering the game of Go with deep neural networks and tree search David Silver et al.

Sub-modules Value Network Mastering the game of Go with deep neural networks and tree search David Silver et al.

Sub-modules Policy Network Value Network Mastering the game of Go with deep neural networks and tree search David Silver et al.

Value Networks Denoising 4 SPP 2^15 SPP

Value Networks Denoising 4 SPP 2^15 SPP Policy Networks Same SPP

Value Networks Denoising 4 SPP 2^15 SPP Policy Networks Same SPP

4 SPP Denoised 32K SPP Ray Tracing 1 sec 2080 Ti 90 mins 12 cores CPU Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation B Xu et al. Siggraph Asia 2019

x Output Input Decoder Encoder loss Ref Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation B Xu et al. Siggraph Asia 2019

L1 VGG Loss L1 VGG Loss + GAN Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation B Xu et al. Siggraph Asia 2019

Diffuse Diffuse x Output Decoder Encoder Input Output loss Specular Specular x Decoder Encoder Input Output Ref Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation B Xu et al. Siggraph Asia 2019

Diffuse Diffuse x Output Decoder Encoder Input Output loss Specular Specular x Decoder Encoder Input Output Ref Auxiliary Albedo, normal, depth

x Element-wise Biasing Conv LeakyReLU Conv Auxiliary

x Element-wise Element-wise Scaling Biasing Conv Conv LeakyReLU LeakyReLU Conv Conv Auxiliary

x Element-wise Element-wise Scaling (AND) Biasing (OR) Conv Conv LeakyReLU LeakyReLU Conv Conv Auxiliary

Denoise comparison 4 SPP Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation B Xu et al. Siggraph Asia 2019

Value Networks Denoising 4 SPP 2^15 SPP Policy Networks Same SPP

Neural Importance Sampling Thomas Müller et al. ACM Transactions on Graphics 2019

incidence radiance map Neural Importance Sampling Thomas Müller et al. ACM Transactions on Graphics 2019

Neural Importance Sampling Thomas Müller et al. ACM Transactions on Graphics 2019

Neural Importance Sampling Thomas Müller et al. ACM Transactions on Graphics 2019

Sub-module End-2-End Model Pictures Differentiable Rendering

Ray Tracing Rasterization Image Centric Object Centric

Visibility Ray Tracing Rasterization Image Centric Object Centric

Shading Ray Tracing Rasterization Image Centric Object Centric

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good NN friendly Great Yes No Enemy

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels 3D Neural Encoder Voxels 32 x 32 x 32 x 16 RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels Visibility 3D Neural Neural 3D-2D Encoder Voxels Pixels 32 x 32 x 32 x 16 32 x 32 x 512 RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels Visibility 3D Neural Neural 3D-2D Encoder Voxels Pixels 32 x 32 x 32 x 16 32 x 32 x 512 RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels Visibility Shading 2D 3D Neural Neural 3D-2D Decoder Encoder Voxels Pixels 32 x 32 x 32 x 16 32 x 32 x 512 MSE pixel loss RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Contour Toon Ambient Occlusion RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

2D 3D Neural Neural 3D-2D Decoder Encoder Voxels Pixels 64 x 64 x 64 x 1 Channel-wise Concatenation Texture Neural or Network Texture Voxels 64 x 64 x 64 x 4 RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Same shape, different textures Same texture, different shapes RenderNet: A deep convolutional network for differentiable rendering from 3D shapes Thu Nguyen-Phuoc et al. NeurIPS 2018

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good NN friendly Great Yes No Enemy

Rasterization a RGB point cloud Neural Point-Based Graphics KA Aliev et al, arxiv 2019

Rasterization a neural point cloud (First three PCA dimensions of the neural descriptor) Neural Point-Based Graphics KA Aliev et al, arxiv 2019

Rasterization a neural point cloud (First three PCA dimensions of the neural descriptor) Neural Point-Based Graphics KA Aliev et al, arxiv 2019

RBG rasterization Neural rasterization Neural Point-Based Graphics KA Aliev et al, arxiv 2019

Deferred Neural Rendering: Neural 3D Mesh Renderer Image Synthesis using Neural Textures H Kato et al, CVPR 2018 J Thies et al, Siggraph 2019

Sub-module End-2-End Model Pictures

?

Approximation Target

Rendered Approximation Target Approximation

Rendered Approximation Target Approximation Loss Back-propagate

Updated Rendered Target Approximation Approximation Loss Back-propagate

Updated Rendered Target Approximation Approximation Loss Back-propagate For Free

Updated Rendered Target Approximation Approximation Expensive Loss Back-propagate

Rendered Target Approximation Decoder Encoder Reconstruction Rendering Loss

Inductive Bias: Separate Appearance from Pose Human perception imposes coordinate frame on objects

Learning 3D representation from natural images without 3D supervision HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

Conditional GANs HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

Conditional GANs Info GANs HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator RenderNet HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator RenderNet 3D StyleGAN HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator RenderNet 3D StyleGAN HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator RenderNet 3D StyleGAN HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator RenderNet HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator RenderNet Real/Fake HoloGAN: Unsupervised learning of 3D representations from natural images Thu Nguyen-Phuoc et al, ICCV 2019