C18 Computer Vision Lecture 8 Recovering 3D from two images II: stereo correspondences, triangulation. Victor Adrian Prisacariu http://www.robots.ox.ac.uk/~victor
Computer Vision: This time… 5. Imaging geometry, camera calibration. 6. Salient feature detection and description. 7. Recovering 3D from two images I: epipolar geometry. 8. 8. Recoveri ring 3D fr from tw two im images II: II: stereo corr rrespondences, tria riangula latio ion, neural l nets. 1. The Correspondence Problem 2. Triangulation for Depth Recovery. 3. 1- views? 4. 2+ views?.
Reconstruction from two views • We have seen that because the projection onto a second camera of the back-projected ray from the first is a straight line, the search for potential matches lies (mostly) across epipolar lines. • Search occurs (mostly ☺ ) in 1D, which must be less demanding than 2D search throughout the image. • We have seen how knowledge of the camera matrices allows us to recover the fundamental matrix, which defines this epipolar geometry.
What next? • The key tasks now are: 1. Determining which points in the images are from the same scene location - the correspondence problem. – We will discuss how to obtain sparse and dense correspondences. 2. Determining the 3D structure by back-projecting rays, or tria triangulation. – Although triangulation seems straightforward, it turns out to be a non-trivial estimation problem to which there are several approaches.
8.1 Establishing Correspondence • With knowledge of the fundamental matrix, we are able to reduce the search for correspondence down to a search along the epipolar line 𝐦′ = 𝑮𝐲 . • Correspondence algorithms can be either: – Sparse: correspondence is sought for individual corner or edge features; or – Dense: correspondences are sought for all the pixels in an image. • We will use both methods.
Sparse Correspondences • Aim to match the points we found in lecture 2 (corners and/or SIFTs). • Outline: 1. Extract image points – same as in L2. 2. Obtain initial corner matches using local descriptors. 3. Remove outliers and estimate the fundamental matrix 𝑮 using RANSAC. 4. Obtain further corner matches using 𝑮 .
Why salient points, especially? • They make the algorithm faster, more efficient and sometimes more robust. • Salient points are: – relatively sparse; – reasonably cheap to compute; – well-localized; – appear quite robustly from frame to frame.
Initial Matching • Extract salient points (SIFT, Harris, etc.) in both images (feature detection). • For each corner 𝐲 in C, make a list of potential matches 𝐲′ in a region in C′ around 𝐲 (heuristic). • Rank the matches by comparing the regions around the corners using e.g. the L2 norm of the SIFT descriptor or cross-correlation (see later). • Process them to reconcile forward-backward inconsistencies. • The idea here is to not to do too much work — just enough to get some good matches.
Initial Matching (example using Harris Corners)
Initial Matching Matches — some good matches, some mismatches. Can still compute 𝑮 with around 50% mismatches. How?
RANSAC – RANdom SAmple Concensus • Suppose you tried to fit a straight line to data containing out outli liers — points which are not properly described by the assumed probability distribution. • The usual methods of least squares are hopelessly corrupted. • Need to detect outliers and exclude them. • Use estimation based on robu obust statis istic ics RANSAC was the first, devised by vision researchers, Fischler & Bolles (1981).
RANSAC algorithm for lines 1. For many repeated trials: 1. Select a random sample of tw two points. 2. Fit a line through them. 3. Count how many other points are within a threshold distance of the line (in inliers). 2. Select the line with the largest number of inliers. 3. Refine the line by fitting it to all the inliers (using least squares). Remarks: • Sample a min inimal set t of f poin ints for your problem (2 for lines). • Repeat such that there is a high chance that at least one minimal set contains only inliers (see tutorial sheet).
RANSAC algorithm for lines
RANSAC algorithm for lines
RANSAC algorithm for 𝑮 1. For many repeated trials: 1. Select a random sample of se seven poin ints. 2. Compute 𝑮 using the 7 point method. 3. Count how many other correspondences are within threshold distance of the epipolar lines (inlie inliers). 2. Select the 𝑮 with the largest number of inliers. 3. Refine 𝑮 by fitting it to all the inliers (using the SVD method).
RANSAC algorithm for 𝑰 1. For many repeated trials: 1. Select a random sample of four points. 2. Compute 𝑰 as in B14. 3. Count how many other correspondences are within threshold distance of the epipolar lines (in inliers). 2. Select the 𝑰 with the largest number of inliers. 3. Refine 𝑰 by fitting it to all the inliers, optimizing the reprojection error: 𝑒 2 𝐲 ′ , 𝑰𝐲 + 𝑒 2 (𝑰 −1 𝐲 ′ , 𝐲) min 𝑰 𝐲,𝐲 ′ ∈inliers
RANSAC Song
Correspondences consistent with epipolar geometry
A Virtual Compass
Fyuse
Dense Correspondences • Aim to match all ll poin ints between the two vie iews. • Useful for e.g. 3D reconstruction, augmented reality, etc. • First we will explore how to adjust the epipolar geometry for convenience a process called image rectification.
Adjusting epipolar geometry: Rectification New optical axes are chosen to be coplanar and perpendicular to the base line, and the new image planes set with the same intrinsic matrix 𝑳 rect .
Rectification • The actual and rectified frames differ by a rotation 𝑺 only. Obviously this must be rotation about the optic centre. • The scene point 𝐘 is projected to 𝐲 in the actual image, and 𝐲 rect in the rectified image. • What is the relationship between them?
Rectification • Allow 𝐘 to be defined in the actual camera frame. The projections are: 𝐲 = 𝑳 𝑱 𝟏 𝐘 = 𝑳𝐘 3×1 and 𝐲 rect = 𝑳 rect 𝑺 𝟏 𝐘 = 𝑳 rect 𝑺𝐘 3×1 • Eliminate 𝐘 3×1 using 𝐘 3×1 = 𝑳 −1 𝐲 (so 𝐘 ’s is actually irrelevant) 𝐲 rect = 𝑳 rect 𝑺𝑳 −1 𝐲 . The 3 × 3 transformation is a homography 𝑰 = 𝑳 rect 𝑺𝑳 −1 - a • plane to plane mapping through the projection center.
A correlation-based approach • The basic assumption here is that the image bands around two corresponding epipolar lines are similar. • The example, for || cameras, shows that this is ~ the case, though there are regions of noise, ambiguity, and occlusion. • In Lecture 2, we saw that auto- correlation provides a method of quantifying the self-similarity of image patches. Here we use cr cross ss-correlati tion.
Zero-Normalized cross-correlation • Images are from different cameras with potentially different global gains and offsets. • Model by 𝐽′ = 𝛽𝐽 + 𝛾 and use zero-normalized cross- correlation. • Step 1: Set source patch 𝐵 around 𝐲 . • Step 2: Subtract the patch mean 𝐵 𝑗𝑘 ← 𝐵 𝑗𝑘 − 𝜈 𝐵 . • Step 3: For each 𝐲′ on the epipolar line: – Generate patch 𝐶 and 𝐶 𝑗𝑘 ← 𝐶 𝑗𝑘 − 𝜈 𝐶 σ 𝑗 σ 𝑘 𝐵 𝑗𝑘 𝐶 𝑗𝑘 – Compute NCC 𝐲, 𝐲 ′ = 2 2 σ 𝑗 σ 𝑘 𝐵 𝑗𝑘 σ 𝑗 σ 𝑘 𝐶 𝑗𝑘 • Useful analogy: Imagine the regions as vectors 𝐵 → 𝐛 and 𝐛⋅𝐜 𝐶 → 𝐜 . . NCC ≡ 𝐛 𝐜 , a scale product of two unit vectors, so − 1 ≤ NCC ≤ 1 .
Examples
Examples Why is cross-correlation a poor measure in Example 2? • The neighbourhood region does not have a distinctive spatial ion → am intensity. Weak aut auto-correlatio ambig iguous cr cross-corr rrela latio ion. • Foreshortening effects perspective distortion. Correlation assumes minim inimal ap appearance ch change which favours fronto-parallel surfaces.
Outline of a dense correspondence algorithm Con onstraints to o ap apply Algorit ithm • uniqueness of match. 1. Rectify images in C and C’. • match ordering. For each pixel 𝐲 in image C: 2. • smoothness of disparity field • Compute NCC for each pixel (disparity gradient limit). 𝐲 ′ along epipolar line 𝐦′ in • figural continuity. image C′. • Choose the match 𝐲′ with the Limit itatio ions highest NCC. • scene must be textured, and • largely fronto-parallel (related Par arameters to o adju adjust to d.g. limit). • size of neighbourhood patches • limit on maximum disparity (𝐲′ − 𝐲)
Example: Left and Right Images
Results: Left Image and 3D Range Map
Example: Full Sequence
Recommend
More recommend
