Computation of the Fundamental Matrix F Shao-Yi Chien Department - PowerPoint PPT Presentation

Two-View Geometry: Computation of the Fundamental Matrix F 簡韶逸 Shao-Yi Chien Department of Electrical Engineering National Taiwan University Fall 2019 1

Outline • Computation of the fundamental matrix F [Slides credit: Marc Pollefeys] 2

Epipolar Geometry: Basic Equation  x' T Fx 0          ' ' ' ' ' ' 0 x xf x yf x f y xf y yf y f xf yf f 11 12 13 21 22 23 31 32 33 separate known from unknown     T ' , ' , ' , ' , ' , ' , , , 1 , , , , , , , , 0 x x x y x y x y y y x y f f f f f f f f f 11 12 13 21 22 23 31 32 33 (data) (unknowns) (linear)   ' ' ' ' ' ' 1 x x x y x y x y y y x y 1 1 1 1 1 1 1 1 1 1 1 1             f 0   ' ' ' ' ' ' 1 x x x y x y x y y y x y   n n n n n n n n n n n n Af  0

The Singularity Constraint rank  detF  Fe   e' T 0 F 2 F 0 0 SVD from linearly computed F matrix (rank 3)   σ 1       σ T σ T σ T σ T F U V U V U V U V  2  1 1 1 2 2 2 3 3 3 σ   3 min F - F' Compute closest rank-2 approximation F   σ 1      σ T σ T σ T F' U V U V U V  2  1 1 1 2 2 2 0   Enforcing singularity!

Effect of enforcing singularity

The Minimum Case – 7 Point Correspondences   ' ' ' ' ' ' 1 x x x y x y x y y y x y 1 1 1 1 1 1 1 1 1 1 1 1             f 0   ' ' ' ' ' ' 1 x x x y x y x y y y x y   7 7 7 7 7 7 7 7 7 7 7 7   A  T σ σ U diag ,..., , 0 , 0 V 7x7 1 7 9x9     A[V V ] 0  T T e.g.V V [ 000000010 ] 8 9 9x2 8     T λF x ( F )x 0, 1...7 i 1 2 i i one parameter family of solutions but F 1 + l F 2 not automatically rank 2

The Minimum Case – Impose Rank 2 (obtain 1 or 3 solutions) F 7pts F F 1 F 2       λF λ λ λ 3 2 det( F ) 0 (cubic equation) a a a a 1 2 3 2 1 0

NOT Normalized 8-point Algorithm   f 11   f   12   f 13     ´ ´ ´ ´ ´ ´ 1 x x y x x x y y y y x y 1 1 1 1 1 1 1 1 1 1 1 1    f  21 ´ ´ ´ ´ ´ ´ 1 x x y x x x y y y y x y     2 2 2 2 2 2 2 2 2 2 2 2  0 f     22              f 23 ´ ´ ´ ´ ´ ´ 1   x x y x x x y y y y x y   n n n n n n n n n n n n f   31 ~10000 ~100 ~10000 ~100 ~10000 ~10000 ~100 ~100 1   f 32   Orders of magnitude difference   f ! 33 Between column of data matrix  least-squares yields poor results

The Normalized 8-point Algorithm Transform image to ~[-1,1]x[-1,1] (0,500) (700,500) (-1,1) (1,1)   2  0 1   700   2    1   500   1 (0,0)     (0,0) (700,0) (-1,-1) (1,-1) Least squares yields good results (Hartley, PAMI ´ 97)

The Normalized 8-point Algorithm Or nomalization with the previous method: (0, 0) 2

The Normalized 8-point Algorithm 11

Geometric Distance • Gold standard • Sampson error • Symmetric epipolar distance

Gold Standard Maximum Likelihood Estimation (= least-squares for Gaussian noise)       2 2  ˆ ˆ ˆ ˆ T x , x x' , x ' subject to x ' F x 0 d d i i i i i Initialize: normalized 8- point, (P,P‘) from F, reconstruct X i Parameterize:   P [I | 0], P' [M | t], X i (overparametrized)   ˆ ˆ x PX , x P' X ’ i i i i Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book)

(Expensive Method) 15

First-order Geometric Error (Sampson Error)   T  e  e e T JJ  1 (one eq./point T e  JJ T scalar) T JJ    x' T Fx 0 e         2 2     2 2 T T T JJ x' F x' F Fx Fx 1 2 1 2 2 represents the square of the j-th entry of the vector Fx i where 𝐺𝑦 𝑗 𝑘 T e e T  x' Fx          2 2    2 2 T T T JJ x' F x' F Fx Fx 1 2 1 2 (problem if some x is located at epipole) advantage: no subsidiary variables required

Symmetric Epipolar Error      2  2 T x' , x x , F x' d F d i i i i i   1 1      T x' Fx            2 2 2 2  T T Fx Fx  x' F x' F  1 2 1 2

Some experiments:

Some experiments:

Some experiments:

Some experiments: Residual error:      2  2 T x' , x x , F x' d F d i i i i i (for all points!)

Recommendations • Do not use unnormalized algorithms • Quick and easy to implement: 8-point normalized • Better: enforce rank-2 constraint during minimization • Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

Automatic Computation of F 1. Interest points 2. Putative correspondences 3. RANSAC 4. Non-linear re-estimation of F 5. Guided matching (repeat 4 and 5 until stable)

Automatic Computation of F 28

Interest Points (e.g.Harris&Stephens ´ 88; Shi&Tomasi ´ 94) Find points that differ as much as possible from all neighboring points homogeneous edge corner

Interest Points Select strongest features (e.g. 1000/image)

Interest Points Evaluate NCC for all features with similar coordinates             e.g. ? ´ w , w h , h x y x x y y 10 10 10 10 Keep mutual best matches ? Still many wrong matches!

Interest Points 3 2 0.96 -0.40 -0.16 -0.39 0.19 4 3 2 -0.05 0.75 -0.47 0.51 0.72 4 1 5 -0.18 -0.39 0.73 0.15 -0.75 1 5 -0.27 0.49 0.16 0.79 0.21 0.08 0.50 -0.45 0.28 0.99 Gives satisfying results for small image motions

RANSAC Step 1. Extract features Step 2. Compute a set of potential matches  Step 3. do (generate Step 3.1 select minimal sample (i.e. 7 matches) hypothesis) Step 3.2 compute solution(s) for F (verify hypothesis) Step 3.3 determine inliers until  (# inliers ,# samples )<95% Step 4. Compute F based on all inliers Step 5. Look for additional matches by guided matching Step 6. Refine F based on all correct matches

RANSAC • Why choose 7-point algorithm instead of 8-point algorithm? • A rank 2 matrix is produced without enforcement • The number of samples that must be tried in order to ensure a high probability of the no outliers is exponential in the size of the sample set • Distance measure       2 2 ˆ ˆ • Reprojection error x , x x' , x ' d d i i i i i T x' Fx  • Sampson approximation         2 2    2 2 T T x' F x' F Fx Fx 1 2 1 2 34

Guided Matching restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line)

geometric relations between two views is fully described by recovered 3x3 matrix F

500 corners 500 corners 37

188 matches 89 outliers 99 inliers 157 matches with guided matching 38

Degenerate Cases • Degenerate cases • Planar scene • Pure rotation • No unique solution • Remaining DOF filled by noise • Use simpler model (e.g. homography) • Model selection (Torr et al., ICCV ´ 98, Kanatani, Akaike) • Compare H and F according to expected residual error (compensate for model complexity)

Image Pair Rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

Planar Rectification (standard approach) ~ image size (calibrated) Distortion minimization Bring two views (uncalibrated) to standard stereo setup (moves epipole to  ) (not possible when in/close to image)

Rectification • Two steps: • Mapping the epipolar to infinity • Finding a projective transformation H of an image that maps the epipole to a point at infinity • Avoid distortion: better to have rigid transformation, to first- order the neighborhood of x 0 may undergo rotation and translation only • Matching transformation • Match the epiplolar lines • Find a match pair that H -T l=H’ -T l’ 42

Mapping the Epipolar to Infinity • For example, given epipole e=(f, 0, 1) T • A good transform is =I if x=y=0 • For an arbitrary x 0 and epipole e • H=GRT: R: rotate to x-axis, T: translate to (f, 0, 1) T 43

Matching Transformation • Target: to minimize • To solve a, b, c, minimize 44

Algorithm Outline 45