Refinement of Surface Mesh for Accurate Multi-View Reconstruction - PowerPoint PPT Presentation

Introduction Depth Map Fusion Mesh Refinement Experiments Refinement of Surface Mesh for Accurate Multi-View Reconstruction International Workshop on Representation and Modeling of Large-scale 3D Environments Asian Conference on Computer Vision Xian, China, September 2009 cek, Radim ˇ Radim Tyleˇ S´ ara { tylecr1|sara } @cmp.felk.cvut.cz Center for Machine Perception, Czech Technical University, Prague cek, R.ˇ 1 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction Depth Map Fusion Reconstruction Pipeline Mesh Refinement Experiments Motivation High resolution images available State-of-the-art MVS results still below accuracy of laser scanners Goal: elimination of sources of inaccuracy imprecise camera calibration variable capture conditions suboptimal representation 3D Photography cek, R.ˇ 2 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction Depth Map Fusion Reconstruction Pipeline Mesh Refinement Experiments Surface Reconstruction Pipeline Pair-wise Corresponding Input images disparity maps regions ⇒ ⇒ ⇓ ⇐ ⇐ Fused depth maps Surface mesh Refined mesh cek, R.ˇ 3 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction The Idea Depth Map Fusion Representation Mesh Refinement Model Refinement Experiments Depth Map Fusion [Tyl09] supporting camera scene 1 supporting camera 4 1−2 2−4 supporting camera 3 fused depth map 2−3 2 pairwise disparity map reference camera Image-based representation with a set of reference cameras Global problem of joint estimation of depths and cameras [Tyl09] R.Tylecek, R.Sara: Depth Map Fusion with Camera Calibration Refinement, CVWW 2009 cek, R.ˇ 4 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction The Idea Depth Map Fusion Representation Mesh Refinement Model Refinement Experiments Why Pair-wise Stereo? Mature methods developed and available [Cech07] Less vulnerable to calibration errors than traditional MVS [Cech07] J.Cech, R.Sara: Efficient sampling of disparity space for fast and accurate matching. BenCOS CVPR 2007 cek, R.ˇ 5 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction The Idea Depth Map Fusion Representation Mesh Refinement Model Refinement Experiments Depth Map Representation Depth maps Registered depth maps ⇒ Back- projection Effective representation natural to input data Complexity linear in the number of Visibility and reference cameras discontinuity maps cek, R.ˇ 6 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction The Idea Depth Map Fusion Representation Mesh Refinement Model Refinement Experiments Model Refinement Model = depth, visibility and discontinuity maps + cameras R j C i − R j C j + ¯ p R j ( R i ) ⊤ ( K i ) − 1 x i λ i p = λ j q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . X ij . . . . . . . . pq . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ♣♣♣♣♣♣♣♣♣♣♣♣♣ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . X i . . . . . . . . . . . . . . ¯ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ♣♣♣♣♣♣♣♣♣♣♣♣♣ p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Camera-depth constraint for each . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ♣♣♣♣♣♣♣♣♣♣♣♣♣ . . correspondence (K-means like) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . λ q . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . φ . . . . . . . . . . . . . . . . . . . . . . .. j . . . . . . . . . . . . . . . . . . . . . . . . . . . Second-order surface model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . λ i . . .. (depth continuity assumption) . . . . . . . . . . . . . ¯ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ( R j ) ⊤ x j . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . One global optimization problem with . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C j . . . . . . . . . . depths ¯ . . . . . . . . . . . . . . . . ( R i ) ⊤ x i . . . . . . . . λ and camera translations C . . . . . . . . . . . . . . . . . . . . . . . p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . as free parameters . . . . . . C i cek, R.ˇ 7 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

Introduction The Idea Depth Map Fusion Photo-consistency Mesh Refinement Contour Matching Experiments Surface Evolution Surface Reconstruction and Refinement Change of representation to triangular mesh Depth maps merged with PSR [Kaz06] Good initial estimate of surface Use of camera calibration refined in previous step Refinement by combined stereo and contour matching for photo-consistency [Kaz06] M. Kazhdan, M. Bolitho and H. Hoppe: Poisson surface reconstruction. Eurographics 2006. cek, R.ˇ 8 / 28 R.Tyleˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement