3D Computer Vision Dmitry Chetverikov, Levente Hajder Eötvös Lóránd University, Faculty of Informatics Chetverikov, Hajder (ELTE IK) 3D Computer Vision 1 / 44
Multi-view reconstruction Principles of multi-view reconstruction 1 Reconstruction for orthogonal and weak-perspective projection 2 Tomasi-Kanade factorization Multi-view perspective reconstruction 3 Concatenation of stereo reconstructions 4 Bundle adjustment 5 Tomasi-Kanade factorization with missing data 6 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 2 / 44
Principles of multi-view reconstruction Outline Principles of multi-view reconstruction 1 Reconstruction for orthogonal and weak-perspective projection 2 Tomasi-Kanade factorization Multi-view perspective reconstruction 3 Concatenation of stereo reconstructions 4 Bundle adjustment 5 Tomasi-Kanade factorization with missing data 6 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 3 / 44
Principles of multi-view reconstruction Possibilities for multi-view reconstruction Concatenation of stereo reconstructions 1 Complicated Reconstruction error cumulated N -view solutons 2 Task is non-linear Difficult soltutions, implementation challenging Reconstrution by simplified camera models 3 Task is linear if orthogonal or weak-perspective projections applied. Chetverikov, Hajder (ELTE IK) 3D Computer Vision 4 / 44
Reconstruction for orthogonal and weak-perspective projection Outline Principles of multi-view reconstruction 1 Reconstruction for orthogonal and weak-perspective projection 2 Tomasi-Kanade factorization Multi-view perspective reconstruction 3 Concatenation of stereo reconstructions 4 Bundle adjustment 5 Tomasi-Kanade factorization with missing data 6 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 5 / 44
Reconstruction for orthogonal and weak-perspective projection Orthogonal projection s i sj r f1 xfi xfj r f2 tf Projection of points � u fp � r T � � f1 = s p − t f (1) r T v fp f2 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 6 / 44
Reconstruction for orthogonal and weak-perspective projection Orthogonal projection si sj r f2 xfi xfj tf r f1 Projection: origin is the center of gravity. � u fp � r T � � f1 = s p (2) r T v fp f2 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 7 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Outline Principles of multi-view reconstruction 1 Reconstruction for orthogonal and weak-perspective projection 2 Tomasi-Kanade factorization Multi-view perspective reconstruction 3 Concatenation of stereo reconstructions 4 Bundle adjustment 5 Tomasi-Kanade factorization with missing data 6 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 8 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Tomasi-Kanade factorization Tracked (matched across multi-frames) coordinates are stacked in measurement matrix W . It can be factorized into two matrices: r T u 11 u 12 · · · u 1 P 11 r T v 11 v 12 · · · v 1 P 12 r T u 21 u 22 · · · u 2 P 21 r T v 21 v 22 · · · v 2 P � � W = = s 1 s 2 . . . s P 22 . . . . ... . . . . . . . . r T · · · u F 1 u F 2 u FP F1 r T v F 1 v F 2 · · · v FP F2 W = MS Chetverikov, Hajder (ELTE IK) 3D Computer Vision 9 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Tomasi-Kanade factorization As W = MS , the rank of W cannot exceed 3 (noiseless-case). Size of M is 2 F × 3 Size of S is 3 × P Lemma: After factorization, the rank cannot inrease Rank reduction of W by Singular Value Decomposition (SVD) Largest 3 singular values/vectors are kept, other ones are reset to zero. W = USV T → W = U ′ S ′ V ′ T σ 1 0 0 0 . . . 0 σ 2 0 0 . . . σ 1 0 0 0 0 0 → S ′ = σ 3 . . . S = 0 σ 2 0 0 0 0 σ 4 . . . 0 0 σ 3 . . . . ... . . . . . . . . Chetverikov, Hajder (ELTE IK) 3D Computer Vision 10 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Ambiguity of factorization Infinite number of solution exist: � MQ − 1 � W = MS = ( QS ) , where Q is a 3 × 3 (affine) matrix. M aff = MQ − 1 : affine motion. S aff = QS affine structure. Constraint to resolve ambiguity: motion vectors r i are orthnormal. Camera motion vectors is of length 1 . 0: r T i1 r i1 = 1 r T i2 r i2 = 1 They are perpendicular to each other: r T i1 r i2 = 0 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 11 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Ambiguity removal Affine → real camera: M aff Q = M m T r T 11 Q 11 m T r T 12 Q 12 . . . . M aff = = . . m T r T F1 Q F1 m T r T F2 Q F2 Constraints for camera vectors: r T m T i1 QQ T m i1 = 1 i1 r i1 = 1 → r T m T i2 QQ T m i2 = 1 i2 r i2 = 1 → r T m T i1 QQ T m i2 = 0 i1 r i2 = 0 → Chetverikov, Hajder (ELTE IK) 3D Computer Vision 12 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Computation of matrix Q Let us introduce the following notation: l 1 l 2 l 3 L = QQ T = l 2 l 4 l 5 l 3 l 5 l 6 Important fact: matrix QQ T is symmetric Constraints can be written in linear form: A i l = b i � � 2 2 2 m i 1 , x 2 m i 1 , x m i 1 , y 2 m i 1 , x m i 1 , z m i 1 , y 2 m i 1 , y m i 1 , z m i 1 , z A i = 2 2 2 m i 2 , x 2 m i 2 , x m i 2 , y 2 m i 2 , x m i 2 , z m i 2 , y 2 m i 2 , y m i 2 , z m i 2 , z m i 1 , x m i 2 , x e 1 e 2 m i 1 , y m i 2 , y m i 1 , y m i 2 , z + m i 2 , y m i 1 , z m i 1 , z m i 2 , z l = [ l 1 , l 2 , l 3 , l 4 , l 5 , l 6 ] T b i = [ 1 , 1 , 0 ] T where m jk , x , m jk , y and m jk , z are the coordinates of vector m jk , and e 1 = m i 1 , x m i 2 , y + m i 2 , x m i 1 , y , e 2 = m i 1 , x m i 2 , z + m i 2 , x m i 2 , z . Chetverikov, Hajder (ELTE IK) 3D Computer Vision 13 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Computation of matrix Q Constraints can be written in linear form: Al = b , � T A T A T A T � A = . . . 1 2 F b = [ 1 , 1 , 0 , 1 , 1 , 0 , . . . , 1 , 1 , 0 ] T Solution by over-determined inhomogeneous linear system of equations Matrix Q can be retrieved from L by SVD: ( SVD ) USU T L = √ Q = U S Chetverikov, Hajder (ELTE IK) 3D Computer Vision 14 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Weak-perspective projection Modified constraints: motion vectors are perpendicular to each other: r T i1 r i2 = 0 Length of vectors are not unit, but equal: r T i1 r i1 = r T i2 r i2 Equations for affine ambuguity, represented by matrix Q as follows: m T i1 QQ T m i1 − m T i2 QQ T m i2 = 0 m T i1 QQ T m i2 = 0 Linear, homogeneous system of equations obtained. Chetverikov, Hajder (ELTE IK) 3D Computer Vision 15 / 44
Reconstruction for orthogonal and weak-perspective projection Tomasi-Kanade factorization Summary of Tomasi-Kanade factorization Tracked points are stacked in measurement matrix W . 1 Origin is moved to the center of gravity, translated coordinates are 2 stacked in matrix ˜ W . SVD computed for ˜ W : ˜ W = USV T . 3 Singular elements are replaced by zero, except the first three 4 values in S : S → S ′ . √ √ S ′ and S aff = S ′ V T . Affine factorization: M aff = U 5 Calculation of matrix Q by metric constraints. 6 Metric factorization: M = M aff Q and S = Q − 1 S aff . 7 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 16 / 44
Multi-view perspective reconstruction Outline Principles of multi-view reconstruction 1 Reconstruction for orthogonal and weak-perspective projection 2 Tomasi-Kanade factorization Multi-view perspective reconstruction 3 Concatenation of stereo reconstructions 4 Bundle adjustment 5 Tomasi-Kanade factorization with missing data 6 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 17 / 44
Multi-view perspective reconstruction Multi-view perspective reconstruction Three-view geometry Extension of epipolar geometry Relationships can be written for 3D points and lines Trifocal tensor introduced as the extension of the fundamental matrix It has small practical impact. Perspective Tomasi-Kanade factorization Problem is a perspective auto-calibration Difficulty: projective depths are different for all point/frames Only iterative solutions exist Very complicated Viable solution: Concatenation of stereo reconstructions Chetverikov, Hajder (ELTE IK) 3D Computer Vision 18 / 44
Concatenation of stereo reconstructions Outline Principles of multi-view reconstruction 1 Reconstruction for orthogonal and weak-perspective projection 2 Tomasi-Kanade factorization Multi-view perspective reconstruction 3 Concatenation of stereo reconstructions 4 Bundle adjustment 5 Tomasi-Kanade factorization with missing data 6 Chetverikov, Hajder (ELTE IK) 3D Computer Vision 19 / 44
Recommend
More recommend
