Towards Complete Free-Form Reconstruction of Complex 3D Scenes from an Unordered Set of Uncalibrated Images R. ˇ H. Cornelius 2 ara 1 D. Martinec 1 T. Pajdla 1 O. Chum 1 S´ J. Matas 1 1 Center for Machine Perception, Czech Technical University, Prague 2 Computational Vision and Active Perception Laboratory, KTH, Stockholm presentation given by Daniel Martinec
Scenario 2/17 1. Targeted imaging — dense high-resolution 3D models
Scenario 2/17 1. Targeted imaging — dense high-resolution 3D models overviews wide baseline calibration detailed views narrow baseline dense matching
Scenario 2/17 1. Targeted imaging — dense high-resolution 3D models overviews wide baseline calibration detailed views narrow baseline dense matching 2. View planning difficult gluing of detailed images difficult
Scenario 2/17 1. Targeted imaging — dense high-resolution 3D models overviews wide baseline calibration detailed views narrow baseline dense matching 2. View planning difficult gluing of detailed images difficult 3. Work for standard imaging too
Overview: Data Processing Pipeline 3/17 1. Maximally stable region matching [Matas, Chum, Urban, Pajdla, BMVC 2002] 2. Estimation of a consistent system of cameras [Martinec, Pajdla, ECCV 2002] [ˇ 3. Dense matching S´ ara, ECCV 2002] [ˇ 4. Local aggregation to fish-scales S´ ara, ICCV 1998]
Step 1: Region Matching 4/17 � stability � discriminability Maximally Stable Extremal Regions (MSER) [Matas, Chum, Urban, Pajdla, BMVC 2002] 1. Match regions across all images 2. Estimate the epipolar geometry for all image pairs
Step 2: Estimation of a Consistent System of Cameras 5/17 . . .
Step 2: Estimation of a Consistent System of Cameras 5/17 Epipolar geometries between image pairs
Step 2: Estimation of a Consistent System of Cameras 5/17 Epipolar geometries between image pairs
Step 2: Estimation of a Consistent System of Cameras 5/17 Goal: reconstruction consistent with all images
Step 2: Estimation of a Consistent System of Cameras 5/17 Goal: reconstruction consistent with all images Problems: 1. occlusions 2. outliers
Problem Formulation 6/17 ← − X 1 x 1 x 2 1 1 p x i = P i λ i Perspective camera projection: X p p � �� � ���� ���� 3 × 4 4 × 1 3 × 1
Problem Formulation 6/17 ← − X 1 x 1 x 2 1 1 p x i = P i λ i Perspective camera projection: X p p � �� � ���� ���� 3 × 4 4 × 1 3 × 1 λ 1 1 x 1 λ 1 2 x 1 λ 1 n x 1 P 1 . . . 1 2 n λ 2 2 x 2 P 2 × × 2 = [ X 1 X 2 . . . X n ] . . . ... . . . . . . � �� � 4 × n λ m 1 x m λ m n x m P m × . . . 1 n � �� � � �� � structure 3 m × 4 R ⇒ rescaled measurement matrix motion
Problem Formulation 6/17 ← − X 1 x 1 x 2 1 1 p x i = P i λ i Perspective camera projection: X p p � �� � ���� ���� 3 × 4 4 × 1 3 × 1 λ 1 1 x 1 λ 1 2 x 1 λ 1 n x 1 P 1 . . . 1 2 n λ 2 2 x 2 P 2 × × 2 = [ X 1 X 2 . . . X n ] . . . ... . . . . . . � �� � 4 × n λ m 1 x m λ m n x m P m × . . . 1 n � �� � � �� � structure 3 m × 4 R ⇒ rescaled measurement matrix motion 1. Estimate λ i p using the epipolar geometry. ⇒ [Sturm, Triggs, ECCV 1996]
Problem Formulation 6/17 ← − X 1 x 1 x 2 1 1 p x i = P i λ i Perspective camera projection: X p p � �� � ���� ���� 3 × 4 4 × 1 3 × 1 λ 1 1 x 1 λ 1 2 x 1 λ 1 n x 1 P 1 . . . 1 2 n λ 2 2 x 2 P 2 × × 2 = [ X 1 X 2 . . . X n ] . . . ... . . . . . . � �� � 4 × n λ m 1 x m λ m n x m P m × . . . 1 n � �� � � �� � structure 3 m × 4 R ⇒ rescaled measurement matrix motion 1. Estimate λ i p using the epipolar geometry. ⇒ [Sturm, Triggs, ECCV 1996] 2. Find basis of R . [Jacobs, CVPR 1997], [Martinec, Pajdla, ECCV 2002] — does not work for big camera movements and sparse data ⇒
Problem Formulation 6/17 ← − X 1 x 1 x 2 1 1 p x i = P i λ i Perspective camera projection: X p p � �� � ���� ���� 3 × 4 4 × 1 3 × 1 λ 1 1 x 1 λ 1 2 x 1 λ 1 n x 1 P 1 . . . 1 2 n λ 2 2 x 2 P 2 × × 2 = [ X 1 X 2 . . . X n ] . . . ... . . . . . . � �� � 4 × n λ m 1 x m λ m n x m P m × . . . 1 n � �� � � �� � structure 3 m × 4 R ⇒ rescaled measurement matrix motion 1. Estimate λ i p using the epipolar geometry. ⇒ [Sturm, Triggs, ECCV 1996] 2. Find basis of R . [Jacobs, CVPR 1997], [Martinec, Pajdla, ECCV 2002] — does not work for big camera movements and sparse data ⇒ 3. [ Fill × and factorize complete R by SVD . ] — no improvement for very sparse data
Optimization & Outlier Removal 7/17 1. oriented-projective reconstruction remove outliers 2. projective BA remove outliers [+iterate] 3. metric update by DIAC 4. metric BA remove outliers [+iterate] 5. metric BA with radial distortion remove outliers [+iterate]
Step 2: Results – Outlier Suppression 8/17
Step 3: Search for dense correspondences 9/17 1. suitable pair identification � epipole outside the image (not necessarily) � #(sparse correspondences) 2. rectification [Gluckman, Nayar, CVPR 2001] 3. dense matching � few mismatches, strict (silent when model not fulfilled) [ˇ � stable matching S´ ara, ECCV 2002]
10/17 � Each disparity map is equivalent to a point cloud − → merge them (2% of points shown) < video >
Step 4: Local Aggregation to Fish-scales 11/17 dense, redundant = ⇒ density reduction Normal vector � piece-wise approximation by simple independent models Center Principal plane � in fact density approximation : mixture of Gaussians � local orientation ∼ covariance of the points
Texturing 12/17 � Use texture from the image contributing most points to the fish-scale
Higher Resolution for Details 13/17 � Close-up image pairs
Head Scene 14/17
Building the Measurement Matrix 15/17 � matches of more reliable EGs are merged sooner � added matches are tested w.r.t. the already merged ones
Building the Measurement Matrix 15/17 � matches of more reliable EGs are merged sooner � added matches are tested w.r.t. the already merged ones = ⇒ works well even when some EGs are absolutely wrong natural joining of overview and detailed images ⇐
Depth Estimation 16/17 � [Sturm, Triggs, ECCV 1996] ( e ij ∧ x i p ) · ( F ij x j p ) λ i λ j p = � e ij ∧ x i p p � 2 c � optimal strategy: connect image pairs with highest #matches ⇐
Valbonne Scene 17/17 14 images [ 768 × 512 ] Outliers 9.15% Mean / maximal reprojection error 0.23 / 0.94 pxl 382 correspondences � �� � 14 images ⇐
Recommend
More recommend
