An Introduction to Holistic 3D Reconstruction Yi Ma EECS - PowerPoint PPT Presentation

An Introduction to Holistic 3D Reconstruction Yi Ma EECS Department, UC Berkeley 1

What is 3D Reconstruction? What is its shape? 2 Image Source: Internet

Applications of 3D Reconstruction 3 Image Source: Internet

Traditional 3D Reconstruction Pipeline Feature Extraction & Matching Multiview Geometry Point Cloud 4 Image Source: Internet

Are Point Clouds Universal Representation? Very Difficult to Storing, Computing, Editing, Visualizing, Interact, and Interpret. A Gear Wheel Scanned by eviXscan 3D Pro Streets Scanned by Velodyne Lidar 5 Image source: Velodyne website Image source: “Diagnostics of machine parts by means of reverse engineering procedures”

Limitation of Traditional 3D Reconstruction Textureless Scenes Reflection/Transparency Repetitive Patterns Multiple Moving Objects Medium/Large Baseline (Correspondence Fail) 6 Image source: Internet

Data-Driven Learning-Based Approaches Voxel Generation Pose Estimation Plane Detection Song, S., et al. (2017) Kehl, Wadim., et al. (2017) Liu, C., et al. (2018) Mesh Generation Depth Map Regression 3D Instance Segmentation Layout Prediction Groueix, T., et al. (2018) Li, Z., & Snavely, N. (2018) Mousavian, A., et al. (2019) Zou, C., et al. (2018) 7

Equivalent to Image Classification? • Recently research [1] suggests deep encoder-decoder networks do not perform reconstruction but classification. For data-driven based depth recovery, DNN is not better (or even worse) than nearest neighbors (NN). [1]. Tatarchenko, Maxim, et al. “What Do Single-view 3D Reconstruction Networks Learn?.” arXiv preprint arXiv:1905.03678 (2019). 8 Ground Truth AtlasNet OGN Matryoshka Clustering Retrieval Oracle NN

Our World is Full of Structures • Man-made environments are rich of structural regularities • straight lines • smooth curves • parallelism • orthogonality • Symmetry • Building code & grammar… 9 Image source: Internet

Importance of Structure in Human 3D Perception • Structure: spatial relationships among multiple points, lines, patches, etc. • Human perceives 3D space by recognizing many types of structure in the scene 10 Lee et al. Geometric Reasoning for Single Image Structure Recovery. CVPR 2009.

Exploring Structures for 3D Reconstruction LOCAL: face-edge-vertex graph, SEMI-GLOBAL: symmetry, GLOBAL: shape grammar smooth curves & surfaces parallelism & orthogonality 11 Image Source: Chen et al., 2007; Pauly et al., 2008 Image Source: https://stuckeman.psu.edu/adapting-modern-architecture-local-context

tures -- What Does a 2D Line Ex Expl ploring ng Local Str truc uctur Drawing Tell Us about the 3D Geometry? “Given a single picture … we usually have definite ideas about the 3-D shapes of objects. To do this we need to use assumptions about the world and the image formation process, since there exist a large number of shapes which can produce the same picture.” -- Takeo Kanade, 1981 [Sinha and Adelson 1993] 12

tures – Some History of Single Line Ex Expl ploring ng Local Str truc uctur Drawing Interpretation • Prototype-based interpretation: Roberts (1965), Falk (1972), Grape (1973) • Polyhedrons: Huffman (1971), Clowes (1971), Mackworth (1973), Sugihara (1978), Whiteley (1979, 1982), Draper (1981), Shapira (1985), … • Curved objects: Turner (1974), Shapira and Freeman (1979), Lee et al. (1985), Malik (1987) … • Paper-made objects (Origami): Kanade (1980, 1981) • Dynamic scenes: Asada et al. (1984) • …… 13

tures – Line Labeling [Huffman- Ex Expl ploring ng Local Str truc uctur Clowes, 1971] • Every line in natural pictures of polyhedron objects should have exactly one of the four labels • Convex (+), concave (-), or occluding ( à , ß ) 14

tures – Junction Dictionary and Ex Expl ploring ng Local Str truc uctur Consistent Labeling [Huffman-Clowes, 1971] • 12 valid configurations for trihedral vertex • L-, Y-, W-types • Represents just 11.5% of all possible configurations • T-junction occurs when an edge occludes another partially. • Does not correspond to a three- dimensional vertex. 15

tures – Junction Dictionary and Ex Expl ploring ng Local Str truc uctur Consistent Labeling [Huffman-Clowes, 1971] • 12 valid configurations for L trihedral vertex W W • L-, Y-, W-types W • Represents just 11.5% of all possible T Y configurations L L • T-junction occurs when an edge occludes another partially. L • Does not correspond to a three- W dimensional vertex. 16

tures – A linear Algebra Approach Ex Expl ploring ng Local Str truc uctur to 3D Reconstruction [Sugihara, 1982] • Consider a picture which is obtained as the orthographic projection of the object • i-th vertex: (x i , y i , z i ) • j-th face: ( a j , b j , c j ) • 3D reconstruction can be formulated as estimating the unknowns z i, a j , b j , c j … 17

tures – A linear Algebra Approach Ex Expl ploring ng Local Str truc uctur to 3D Reconstruction [Sugihara, 1982] Line label assignments to the picture provide two forms of constraints: 1. Vertex i should be on the j-th face: 2. Vertex t should be nearer than the k-th face 18

tures – A linear Algebra Approach Ex Expl ploring ng Local Str truc uctur to 3D Reconstruction [Sugihara, 1982] Theorem: A labeled line drawing represents a polyhedral scene if and only if the linear system has a solution. In practice, there are usually infinite number of solutions … 19

tures – Shape Recovery via Ex Expl ploring ng Local Str truc uctur Optimization • To resolve ambiguity, one option is to use additional cues such as shading [Sugihara, 1986] : Lambertian surface with light source direction l: 20

tures – Shape Recovery via Ex Expl ploring ng Local Str truc uctur Optimization • Another option is to invoke additional structural priors, such as smoothness and regularity : “To interpret a polygon in the image, we try to find a configuration of the vertices in space that makes the three-dimensional figure as regular as possible . Regularity might be measured in a e.g., f(w) = “sum of the squares of variety of ways … we prefer local features angles of faces” which are more likely to survive occlusion.” -- Barrow and Tenenbaum, 1981 21

tures – Some History of Additional Ex Expl ploring ng Local Str truc uctur Cues in Single Line Drawing Interpretation • Light intensity: Horn (1975), Woodham (1977, 1981), Ikeuchi (1981) Coleman and Jain (1982) • Apparent distortion of known patterns: Kender (1979), Kanade (1981), Ikeuchi (1984) • Surface contour: Stevens (1981), Barrow and Tenenbaum (1981), Marr (1982) • Texture: Bajcsy and Lieberman (1976), Witkin (1981) • Vanishing points: Nakatani and Kitahashi (1984) • …… 22

Exploring Structures for 3D Reconstruction LOCAL: face-edge-vertex graph, SEMI-GLOBAL: symmetry, GLOBAL: shape grammar smooth curves & surfaces parallelism & orthogonality 23 Image Source: Chen et al., 2007; Pauly et al., 2008 Image Source: https://stuckeman.psu.edu/adapting-modern-architecture-local-context

Sym ymmetry y Structures Symmetry captures almost all “ regularities ” . 24

Sym ymmetry y Structures – Hidden Images from Rotation 25

Sym ymmetry y Structures – Hidden Images from Reflection 26

Sym ymmetry y Structures – Hidden Images from Translation 27

Sym ymmetric Structure & Group Definition. A set of 3-D features S is called a symmetric structure if there exists a non-trivial subgroup G of E(3) that acts on it such that for every g in G, the map is an (isometric) automorphism of S. We say the structure S has a group symmetry G. 28

Sym ymmetry y Structures – Hidden Multiple Views 29

Sym ymmetry y Structures – Hidden Multiple Views Solving g 0 from Lyapunov equations: with g i ’ and g i known. 30

Sym ymmetry y Structures 3-D reconstruction with symmetry is simple, accurate and robust! 31

Sym ymmetry y Structures – Hidden Images in Each View Symmetry on object 2 (1) 1 (2) 4 (3) 3 (4) Virtual camera-camera 32

Sym ymmetry y Structures – Reflective Homography 2 pairs of symmetric points 2 (1) 1 (2) Reflective homography 4 (3) 3 (4) Decompose H to obtain ( R’, T’, N ) and T 0 Solve Lyapunov equation to obtain R 0 . 33

Sym ymmetry y Structures – Alignment of Different Objects ? 34

Sym ymmetry y Structures – Scale Correction For a point p on the intersection line ? 35

Sym ymmetry y Structures – Alignment of Different Views z z z 36

Sym ymmetry y Structures – Scale Correction For any image x 1 in the first view, its corresponding image in the second view is: 37

Sym ymmetry y Structures – Alignment of Multiple Views Method is object-centered and baseline-independent. 38

Sym ymmetry y Structures – Experiment Results 39

Sym ymmetry y Structures – Experiment Results 40

Sym ymmetry y Structures – Image Transfer 41

Sym ymmetry y Structures – Camera Pose 42

Sym ymmetry y Structures – Full 3D Model 43