Neural l Renderin ing Prof. Leal-Taix and Prof. Niessner 1 - PowerPoint PPT Presentation

Neural l Renderin ing Prof. Leal-Taixé and Prof. Niessner 1

Renderin ing Camera Def. - Intrinsics - Often: 3D Scene: - focal length - Material - principal point) - Lighting - Geometry (incl. animation) Camera View Point - Extrinsics - 6 DoF (3rot, 3trans) Prof. Leal-Taixé and Prof. Niessner 2

Photo-reali listic Im Image Synthesis is The Rendering Equation [Kajiya 86] Prof. Leal-Taixé and Prof. Niessner 3

Need 3D Content fo for r Renderi ring Geometry Textures Material & Lighting Prof. Leal-Taixé and Prof. Niessner 4

Computer Vis isio ion fo for r Reconstructio ion Prof. Leal-Taixé and Prof. Niessner 5 ICCV’09 [Agarwal et al.]: Building Rome in a Day

3D Dig igit itization Computer Graphics Computer Vision Prof. Leal-Taixé and Prof. Niessner 6

Tra raditi itional Gra raphic ics vs Deep Learn rnin ing Generative Adversarial Networks 3D Model + Textures + Shading -> Synthetic Image [Karras et al. 18] Discriminator loss Generator loss Star Wars Rogue One Prof. Leal-Taixé and Prof. Niessner 7

Id Idea of f Neural Renderin ing Novel View point synthesis: Neural Network -> Encodes entire 6 DoF Camera scene description, Pose / View Point lighting, materials, etc. Prof. Leal-Taixé and Prof. Niessner 8

Neural Renderin ing wit ith Pix ix2Pix ix Ground truth for training - Pose + Target Image (e.g., observed from real world) - Constrain with re-rendering loss Testing - Given unseen pose, generate image Prof. Leal-Taixé and Prof. Niessner 9

Neural Renderin ing wit ith Pix ix2Pix ix Prof. Leal-Taixé and Prof. Niessner 10

Other r Neural Renderin ing - Conditioned on Faces (Deep Video Portraits) - Conditioned on Human Skeleton (Everybody Dance Now) Prof. Leal-Taixé and Prof. Niessner 11

Neural Renderin ing wit ith Pix ix2Pix ix Prof. Leal-Taixé and Prof. Niessner 12

Deep Voxels Prof. Leal-Taixé and Prof. Niessner 13 [Sitzmann et al. CVPR’19] Deep Voxels

Deep Voxels • Main idea for video generation: – Why learn 3D operations with 2D Convs !?!? – We know how 3D transformations work • E.g., 6 DoF rigid pose [ [ R | t | t ] – Incorporate these into the architectures • Need to be differentiable! – Example application: novel view point synthesis • Given rigid pose, generate image for that view Prof. Leal-Taixé and Prof. Niessner 14 [Sitzmann et al. CVPR’19] Deep Voxels

Deep Voxels 2D Feature 3D Features Renderin Extraction g Projection Layer Lifting Layer 3D 2D 2D 3D Source Output Target Source 2D U-Net 2D U-Net 3D U-Net View R, t View R, t Simplified overview for novel view synthesis Prof. Leal-Taixé and Prof. Niessner 15 [Sitzmann et al. CVPR’19] Deep Voxels

Deep Voxels Prof. Leal-Taixé and Prof. Niessner 16 [Sitzmann et al. CVPR’19] Deep Voxels

Deep Voxels Occlusion Network: Issue: we don’t know the depth for the target! -> Per-pixel softmax along the ray -> Network learns the depth Prof. Leal-Taixé and Prof. Niessner 17 [Sitzmann et al. CVPR’19] Deep Voxels

Deep Voxels Prof. Leal-Taixé and Prof. Niessner 18 [Sitzmann et al. ’18] Deep Voxels

Deep Voxels Prof. Leal-Taixé and Prof. Niessner 19 [Sitzmann et al. ’18] Deep Voxels

Deep Voxels: : In Insights • Lifting from 2D to 3D works great – No need to take specific care for temp. coherency! • All 3D operations are differentiable • Currently, only for novel view-point synthesis – I.e., cGAN for new pose in a given scene • But: limited resolution due to dense 3D voxel grid Prof. Leal-Taixé and Prof. Niessner 20 [Sitzmann et al. ’18] Deep Voxels

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Both Scene Representation and Differentiable Renderer often adapted from traditional computer graphics. Prof. Leal-Taixé and Prof. Niessner 21 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Pros Pros Cons Cons Prof. Leal-Taixé and Prof. Niessner 22 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Pros Pros Cons Cons Prof. Leal-Taixé and Prof. Niessner 23 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Importing 3D stru Im ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering Pros Pros High quality Generalizes Only 2.5D Cons Cons Size Prof. Leal-Taixé and Prof. Niessner 24 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Importing 3D stru Im ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering Pros Pros High quality Generalizes Only 2.5D Cons Cons Size Prof. Leal-Taixé and Prof. Niessner 25 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering “True 3D” Pros Pros High quality High quality Generalizes No reconstruction Only 2.5D Cons Cons priors Size Memory O(n 3 ) Prof. Leal-Taixé and Prof. Niessner 26 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering “True 3D” Pros Pros High quality High quality Generalizes No reconstruction Only 2.5D Cons Cons priors Size Memory O(n 3 ) Prof. Leal-Taixé and Prof. Niessner 27 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering “True 3D” High quality Pros Pros High quality High quality Generalizes No reconstruction Requires good SFM Only 2.5D Cons Cons priors No compact Size Memory O(n 3 ) representation Prof. Leal-Taixé and Prof. Niessner 28 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering “True 3D” High quality Pros Pros High quality High quality Generalizes No reconstruction Requires good SFM Only 2.5D Cons Cons priors No compact Size Memory O(n 3 ) representation Prof. Leal-Taixé and Prof. Niessner 29 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering “True 3D” High quality High quality Pros Pros High quality High quality Generalizes Requires good SFM No reconstruction Requires good SFM Only 2.5D Cons Cons priors No compact Size Memory O(n 3 ) representation Prof. Leal-Taixé and Prof. Niessner 30 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)

Im Importing 3D stru ructure fr from CG Scene Scene Representa tion tion Multi-Plane Images Voxelgrids Image-based Implicit Function Point Clouds (Alpha) compositing Volumetric Sphere-Traced Rasterization Splatting Rendere rer Ray-based Volumetric Fast rendering “True 3D” High quality High quality Pros Pros High quality High quality Generalizes Requires good SFM No reconstruction Requires good SFM Only 2.5D Cons Cons priors No compact Size Memory O(n 3 ) representation Prof. Leal-Taixé and Prof. Niessner 31 Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)