deep models for 3d reconstruction
play

Deep Models for 3D Reconstruction Andreas Geiger Autonomous Vision - PowerPoint PPT Presentation

Jan 21, 2024 •513 likes •1.22k views

Deep Models for 3D Reconstruction Andreas Geiger Autonomous Vision Group, MPI for Intelligent Systems, T ubingen Computer Vision and Geometry Group, ETH Z urich October 12, 2017 Max Planck Institute for Intelligent Systems Autonomous

  1. Deep Models for 3D Reconstruction Andreas Geiger Autonomous Vision Group, MPI for Intelligent Systems, T¨ ubingen Computer Vision and Geometry Group, ETH Z¨ urich October 12, 2017 Max Planck Institute for Intelligent Systems Autonomous Vision Group

  2. 3D Reconstruction [Furukawa & Hernandez: Multi-View Stereo: A Tutorial] Task: ◮ Given a set of 2D images ◮ Reconstruct 3D shape of object/scene 2

  3. 3D Reconstruction Pipeline Input Images 3

  4. 3D Reconstruction Pipeline Input Images Camera Poses 3

  5. 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences 3

  6. 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences Depth Maps 3

  7. 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences Depth Map Fusion Depth Maps 3

  8. 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences 3D Reconstruction Depth Map Fusion Depth Maps 3

  9. 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences 3D Reconstruction Depth Map Fusion Depth Maps 3

  10. Large 3D Datasets and Repositories [Newcombe et al ., 2011] [Choi et al ., 2011] [Dai et al ., 2017] [Wu et al ., 2015] [Chang et al ., 2015] [Chang et al ., 2017] 4

  11. Can we learn 3D Reconstruction from Data?

  12. OctNet: Learning Deep 3D Representations at High Resolutions [Riegler, Ulusoy, & Geiger, CVPR 2017]

  13. Deep Learning in 2D [LeCun, 1998] 7

  14. Deep Learning in 3D 8

  15. Deep Learning in 3D ◮ Existing 3D networks limited to ∼ 32 3 voxels 8

  16. 3D Data is often Sparse [Geiger et al ., 2012] 9

  17. 3D Data is often Sparse [Li et al ., 2016] 9

  18. 3D Data is often Sparse [Li et al ., 2016] Can we exploit sparsity for efficient deep learning? 9

  19. Network Activations Layer 1: 32 3 Layer 2: 16 3 Layer 3: 8 3 10

  20. Network Activations Layer 1: 32 3 Layer 2: 16 3 Layer 3: 8 3 10

  21. Network Activations Layer 1: 32 3 Layer 2: 16 3 Layer 3: 8 3 Idea: ◮ Partition space adaptively based on sparse input 10

  22. Convolution 11

  23. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  24. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  25. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  26. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  27. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  28. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  29. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  30. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  31. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  32. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  33. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  34. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  35. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  36. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  37. 0.125 0.250 0.125 0.000 0.000 0.000 0.125 0.250 0.125 Convolution 11

  38. Convolution 11

  39. Convolution 11

  40. Convolution ◮ Differentiable ⇒ allows for end-to-end learning 11

  41. Efficient Convolution This operation can be implemented very efficiently: ◮ 4 different cases ◮ First case requires only 1 evaluation! 12

  42. Pooling 13

  43. Pooling 13

  44. Pooling 13

  45. Pooling ◮ Unpooling operation defined similarly 13

  46. Results: 3D Shape Classification Airplane Convolution Convolution Fully Fully and Pooling and Pooling Conn. Conn. 14

  47. Results: 3D Shape Classification 80 OctNet 70 DenseNet Memory [GB] 60 50 40 30 20 10 0 8 3 16 3 32 3 64 3 128 3 256 3 Input Resolution 15

  48. Results: 3D Shape Classification 16 OctNet 14 DenseNet 12 Runtime [s] 10 8 6 4 2 0 8 3 16 3 32 3 64 3 128 3 256 3 Input Resolution 15

  49. Results: 3D Shape Classification 0 . 95 0 . 90 Accuracy 0 . 85 0 . 80 0 . 75 OctNet DenseNet 0 . 70 8 3 16 3 32 3 64 3 128 3 256 3 Input Resolution ◮ Input: voxelized meshes from ModelNet 16

  50. Results: 3D Shape Classification OctNet 1 0 . 94 OctNet 2 0 . 92 Accuracy OctNet 3 0 . 90 0 . 88 0 . 86 8 3 16 3 32 3 64 3 128 3 256 3 Input Resolution ◮ Input: voxelized meshes from ModelNet 16

  51. Results: 3D Shape Classification 17

  52. Results: 3D Semantic Labeling Input Prediction ◮ Dataset: RueMonge2014 18

  53. Results: 3D Semantic Labeling Skip Skip Convolution Convolution Unpooling Unpooling and Pooling and Pooling and Conv. and Conv. ◮ Decoder octree structure copied from encoder 19

  54. Results: 3D Semantic Labeling IoU [Riemenschneider et al ., 2014] 42.3 [Martinovic et al ., 2015] 52.2 [Gadde et al ., 2016] 54.4 OctNet 64 3 45.6 OctNet 128 3 50.4 OctNet 256 3 59.2 20

  55. OctNetFusion: Learning Depth Fusion from Data [Riegler, Ulusoy, Bischof & Geiger, 3DV 2017]

  56. Volumetric Fusion w ( p ) ˆ w i ( p ) d i ( p ) + ˆ d ( p ) d i +1 ( p ) = ◮ p ∈ R 3 : voxel location w i ( p ) + ˆ w ( p ) ◮ d : distance, w : weight w i +1 ( p ) = w i ( p ) + ˆ w ( p ) [Curless and Levoy, SIGGRAPH 1996] 22

  57. Volumetric Fusion ◮ Pros : ◮ Simple, fast, easy to implement ◮ Defacto ”gold standard” (KinectFusion, Voxel Hashing, ...) Ground Truth Volumetric Fusion 23

  58. Volumetric Fusion ◮ Pros : ◮ Simple, fast, easy to implement ◮ Defacto ”gold standard” (KinectFusion, Voxel Hashing, ...) ◮ Cons : ◮ Requires many redundant views to reduce noise ◮ Can’t handle outliers / complete missing surfaces Ground Truth Volumetric Fusion 23

  59. TV-L1 Fusion ◮ Pros : ◮ Prior on surface area ◮ Noise reduction Ground Truth Volumetric Fusion TV-L1 Fusion 23

  60. TV-L1 Fusion ◮ Pros : ◮ Prior on surface area ◮ Noise reduction ◮ Cons : ◮ Simplistic local prior (penalizes surface area, shrinking bias) ◮ Can’t complete missing surfaces Ground Truth Volumetric Fusion TV-L1 Fusion 23

  61. Learned Fusion ◮ Pros : ◮ Learn noise suppression from data ◮ Learn surface completion from data Ground Truth Volumetric Fusion TV-L1 Fusion OctNetFusion 23

  62. Learned Fusion ◮ Pros : ◮ Learn noise suppression from data ◮ Learn surface completion from data ◮ Cons : ◮ Requires large 3D datasets for training ◮ How to scale to high resolutions? Ground Truth Volumetric Fusion TV-L1 Fusion OctNetFusion 23

  63. Learning 3D Fusion Skip Skip Convolution Convolution Unpooling Unpooling and Pooling and Pooling and Conv. and Conv. Input Representation: Output Representation: ◮ TSDF ◮ Occupancy ◮ Higher-order statistics ◮ TSDF 24

  64. Learning 3D Fusion Skip Skip Convolution Convolution Unpooling Unpooling and Pooling and Pooling and Conv. and Conv. What is the problem? 24

  65. Learning 3D Fusion Skip Skip Convolution Convolution Unpooling Unpooling and Pooling and Pooling and Conv. and Conv. What is the problem? ◮ Octree structure unknown ⇒ needs to be inferred as well! 24

  66. OctNetFusion Architecture 64³ 64³ ∆ 64 Octree Structure Input Output Features 128³ 128³ ∆ 128 Octree Structure Input Output Features 256³ 256³ ∆ 256 Input Output 25

  67. Results: Surface Reconstruction VolFus TV-L1 Ours Ground Truth 64 3 128 3 256 3 26

  68. Results: Volumetric Completion [Firman, 2016] Ours Ground Truth 27

  69. Thank you!

Recommend

Addressing The False Negative Problem of Deep Learning MRI Reconstruction Models by Paper #28

Addressing The False Negative Problem of Deep Learning MRI Reconstruction Models by Paper #28

Addressing The False Negative Problem of Deep Learning MRI Reconstruction Models by Paper #28 Adversarial Attacks and Robust Training Kaiyang Cheng*, Francesco Caliv*, Rutwik Shah, Misung Han, Sharmila Majumdar, Valentina Pedoia Disclosure

286 views • 25 slides
3D RECONSTRUCTION Reconstruction method Reconstruction from images Reconstruction from video

3D RECONSTRUCTION Reconstruction method Reconstruction from images Reconstruction from video

3D RECONSTRUCTION Reconstruction method Reconstruction from images Reconstruction from video Using Kinect Raw Depth Image Infrared laser projector Monochrome CMOS sensor Demo Kinect Raw data Real-time Reconstruction Pipeline Measurement

508 views • 34 slides
Updates on Vertex Reconstruction with Deep Learning in Pandora J. Ahmed. Second Year PhD Student.

Updates on Vertex Reconstruction with Deep Learning in Pandora J. Ahmed. Second Year PhD Student.

DUNE UK Meeting 11th of December 2019 Updates on Vertex Reconstruction with Deep Learning in Pandora J. Ahmed. Second Year PhD Student. Working on Pandora reconstruction Focusing on DUNE FD and ProtoDUNE. Supervisor: Dr John Marshall 1

549 views • 27 slides
GradNet: Unsupervised Deep Screened Poisson Reconstruction for GradientDomain Rendering Jie Guo

GradNet: Unsupervised Deep Screened Poisson Reconstruction for GradientDomain Rendering Jie Guo

GradNet: Unsupervised Deep Screened Poisson Reconstruction for GradientDomain Rendering Jie Guo 1 Mengtian Li 1 Quewei Li 1 Yuting Qiang 1 Bingyang Hu 1 Yanwen Guo 1 LingQi Yan 2 1 State Key Lab for Novel Software 2 University of California,

963 views • 49 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
Applying Deep Neural Network Techniques for LArTPC Data Reconstruction Laura Domine (Stanford

Applying Deep Neural Network Techniques for LArTPC Data Reconstruction Laura Domine (Stanford

Applying Deep Neural Network Techniques for LArTPC Data Reconstruction Laura Domine (Stanford University / SLAC) Fermilab Machine Learning Group meeting - 11/7/18 Plan 1. LArTPC & Deep Learning 2. Examples of applications: UResNet &

585 views • 39 slides
Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning Yuchi Huo, Rui Wnag, Ruzhang Zheng, Hualin Xu, Hujun Bao, Sung-eui Yoon KAIST and CAD&CG Global Illumination Path Guiding Filtering

257 views • 23 slides
Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting Wang* 2 , Quanshi Zhang 1 , Runjin Chen 1 , Di He 2 , Xing Xie 2 * Equal Contribution 1 John Hopcroft Center and the MoE Key Lab of Artificial

219 views • 11 slides
RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab

RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab

RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab luzhc@dgp.toronto.edu 1 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng and Christopher Potts. Recursive Deep Models for Semantic

518 views • 29 slides
Deep Neural Network Based Frame Reconstruction For Optimized Video Coding - An AV2 Approach

Deep Neural Network Based Frame Reconstruction For Optimized Video Coding - An AV2 Approach

Deep Neural Network Based Frame Reconstruction For Optimized Video Coding - An AV2 Approach Dandan Ding Hangzhou Normal University Background of our project 01 AV1 is the most advanced standardized codec available today. Research and

474 views • 22 slides
Reconstruction and Repair of 3D Surfaces TalkID 23152 This session will give the audience a quick

Reconstruction and Repair of 3D Surfaces TalkID 23152 This session will give the audience a quick

government-funded by supervised by Deep 3D Machine Learning for Reconstruction and Repair of 3D Surfaces TalkID 23152 This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep

519 views • 28 slides
and Deep Reconstruction Dr. Uwe Kruger Department of Biomedical Engineering Jonsson Engineering

and Deep Reconstruction Dr. Uwe Kruger Department of Biomedical Engineering Jonsson Engineering

Projection-based Chemometrics and Deep Reconstruction Dr. Uwe Kruger Department of Biomedical Engineering Jonsson Engineering Center Rensselaer Polytechnic Institute Presentation Outline Motivation for kernel-based methods (kernel density

850 views • 38 slides
MODELS OF VOLCANIC ASH FIELDS RECONSTRUCTION Raputa V.F., Yaroslavtseva T.V. Institute of

MODELS OF VOLCANIC ASH FIELDS RECONSTRUCTION Raputa V.F., Yaroslavtseva T.V. Institute of

MODELS OF VOLCANIC ASH FIELDS RECONSTRUCTION Raputa V.F., Yaroslavtseva T.V. Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk Vulcan Puyeue (Chile), June 2011 1. Experimental studies Fig. 1. Paramushir

391 views • 19 slides
BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For Group Activity Recognition Deng et al. BMVC 2015 Simon Fraser University, SPORTLOGIQ , Canada Overview 25 April 2016 Deep Structured Models For

701 views • 40 slides
Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant recognition in the ventral stream Center Name Presenter Name Towards a theory of the above tomaso poggio, CBMM, BCS, CSAIL, McGovern MIT Plan

1.14k views • 81 slides
Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-10-04 Roadmap Challenges of Unstructured Modeling Using Graphs to Describe Model Structure

974 views • 34 slides
Deep Single-View 3D Object Reconstruction with Visual Hull Embedding 1,2 2 1 2 Hanqing Wang,

Deep Single-View 3D Object Reconstruction with Visual Hull Embedding 1,2 2 1 2 Hanqing Wang,

Deep Single-View 3D Object Reconstruction with Visual Hull Embedding 1,2 2 1 2 Hanqing Wang, Jiaolong Yang, Wei Liang, Xin Tong 2 1 Beijing Institute of Technology Microsoft Research Asia Beijing, China

789 views • 27 slides
Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

A New Framework For Hybrid Models An Application: Deep Hybrid Models Supervised and Semi-Supervised Experiments Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and Stefano Ermon Department of Computer

523 views • 29 slides
Improving Robustness of Deep-Learning-Based Image Reconstruction Ankit Raj [1] , Yoram Bresler

Improving Robustness of Deep-Learning-Based Image Reconstruction Ankit Raj [1] , Yoram Bresler

Improving Robustness of Deep-Learning-Based Image Reconstruction Ankit Raj [1] , Yoram Bresler [1] , Bo Li [2] [1] Department of ECE, [2] Department of CS University of Illinois at Urbana-Champaign Acknowledgment - This work was supported in

410 views • 22 slides
Spectral Reconstruction with Deep Neural Networks Lukas Kades Cold Quantum Coffee - Heidelberg

Spectral Reconstruction with Deep Neural Networks Lukas Kades Cold Quantum Coffee - Heidelberg

Spectral Reconstruction with Deep Neural Networks Lukas Kades Cold Quantum Coffee - Heidelberg University - arXiv: 1905.04305, Lukas Kades, Jan M. Pawlowski, Alexander Rothkopf, Manuel Scherzer, Julian M. Urban, Sebastian J. Wetzel, Nicolas

713 views • 28 slides
Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models have shown promising results in many domains. However, such models often have millions of parameters. The deep learning models face the following

314 views • 16 slides
Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye

Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye

Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan *equal contribution Predictive Models Hybrid Models with Deep and Invertible Features Predictive Models

809 views • 16 slides
Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation Outline Exploring Context with Deep Structured Models Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel; Efficient Piecewise Training

992 views • 45 slides
Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

757 views • 22 slides

More recommend