Lecture 18: Multi-view reconstruction 1 Announcements If you - PowerPoint PPT Presentation

Lecture 18: Multi-view reconstruction 1

Announcements • If you have a question: just ask it (or send message saying that you have a question) • Please send us feedback! 
 • PS8 out: representation learning • Final presentation will take place over video chat. - We’ll send a sign-up sheet next week 2

Today • Finding correspondences • RANSAC • Structure from motion 3

Motivating example: panoramas 4 Source: N. Snavely

Warping with a homography Need correspondences! 5

Local features: main components Detection: Identify the interest points 1) 2) Description : Extract vector feature ( 1 ) ( 1 ) … x [ x , , x ] = descriptor surrounding each interest point. 1 1 d 3) Matching: Determine correspondence ( 2 ) ( 2 ) … x [ x , , x ] = between descriptors in two views 2 1 d 6 Source: K. Grauman

Which features should we match? • How does the window change when you shift it? • Shifting the window in any direction causes a big change “corner”: 
 “flat” region: 
 “edge”: significant change in no change in all no change along the all directions directions edge direction 7 Source: S. Seitz, D. Frolova, D. Simakov, N. Snavely

Finding keypoints Compute difference-of-Gaussians Find local optima in space/ filter (approx. to Laplacian) scale using pyramid 8

Feature descriptors We know how to detect good points Next question: How to match them? ? Answer: Come up with a descriptor for each point, find similar descriptors between the two images 9 Source: N. Snavely

Simple idea: normalized image patch Take 40x40 window around feature • Find dominant orientation 40 pixels 8 pixels • Rotate to horizontal • Sample 8x8 square window centered at feature • Intensity normalize the window by subtracting the mean, dividing by the standard deviation in the window CSE 576: Computer Vision 10 Source: N. Snavely, M. Brown

Scale Invariant Feature Transform Basic idea: hand-crafted CNN • Take 16x16 square window around detected feature • Compute edge orientation for each pixel • Create histogram of edge orientations 2 π 0 angle histogram 11 Source: N. Snavely, D. Lowe

Scale Invariant Feature Transform Create the descriptor: • Rotation invariance: rotate by “dominant” orientation • Spatial invariance: spatial pool to 2x2 • Compute an orientation histogram for each cell • 16 cells * 8 orientations = 128 dimensional descriptor 12 Source: N. Snavely, D. Lowe

SIFT invariances 13 Source: N. Snavely

Which features match? 14 Source: N. Snavely

Finding matches How do we know if two features match? Simple approach: are they the nearest neighbor in L 2 distance, ||f 1 - f 2 || – Can give good scores to ambiguous (incorrect) matches – f 1 f 2 I 1 I 2 15 Source: N. Snavely

Finding matches Add extra tests: Ratio distance = ||f 1 - f 2 || / || f 1 - f 2 ’ || • f 2 is best SSD match to f 1 in I 2 • f 2 ’ is 2 nd best SSD match to f 1 in I 2 • Forward-backward consistency: f 1 should also be nearest neighbor of f 2 • f 1 f 2' f 2 I 1 I 2 16 Source: N. Snavely

Feature matching example 51 feature matches after ratio test 17 Source: N. Snavely

Feature matching example 58 feature matches after ratio test 18 Source: N. Snavely

From matches to homography (x 1 ,y 1 ) (x’ 1 ,y’ 1 ) x 1 a b c x 1 ’ . = d e f y 1 y 1 ’ g h i 1 w 1 19 Source: Torralba, Isola, Freeman

From matches to homography Point in 1st image Matched point in 2nd X i || 2 || f H ( p i ) − p 0 J ( H ) = minimize i f H ( p i ) = Hp i / ( H T where applies homography 3 p i ) • Plug into nonlinear least squares solver and solve! • Can also use robust loss (e.g. L 1 ) • Can be slow 20

Direct linear transform x 1 a b c x 1 ’ . = d e f y 1 y 1 ’ g h i 1 w 1 Going to heterogeneous coordinates: ax 1 + by 1 +c x 1 ’= gx 1 + hy 1 +i dx 1 + ey 1 +f y 1 ’= gx 1 + hy 1 +i Re-arranging the terms: gx 1 x’ 1 + hy 1 x’ 1 +ix 1 = ax 1 + by 1 +c gx 1 y’ 1 + hy 1 y’ 1 +ix 1 = dx 1 + ey 1 +f 21 Source: Torralba, Freeman, Isola

Direct linear transform gx 1 x’ 1 + hy 1 x’ 1 +ix 1 = ax 1 + by 1 +c gx 1 y’ 1 + hy 1 y’ 1 +ix 1 = dx 1 + ey 1 +f Re-arranging the terms: gx 1 x’ 1 + hy 1 x’ 1 +ix’ 1 - ax 1 - by 1 - c = 0 gx 1 y’ 1 + hy 1 y’ 1 +iy’ 1 - dx 1 - ey 1 - f = 0 In matrix form. Can solve using Singular Value Decomposition (SVD). 0 
 -x 1 -y 1 -1 0 0 0 x 1 x’ 1 y 1 x’ 1 x’ 1 a 
 b 
 = 0 c 
 0 0 0 -x 1 -y 1 -1 x 1 y’ 1 y 1 y’ 1 y’ 1 d 
 e 
 f 
 g 
 h 
 i Fast to solve (but not using “right” loss function). Uses an algebraic trick. Often used in practice for initial solutions! 22 Source: Torralba, Freeman, Isola

Outliers outliers inliers 23 Source: N. Snavely

Robustness • Let’s consider the problem of linear regression Problem: Fit a line to these data points Least squares fit • How can we fix this? 24 Source: N. Snavely

Counting inliers 25 Source: N. Snavely

Counting inliers Inliers: 3 26 Source: N. Snavely

Counting inliers Inliers: 20 27 Source: N. Snavely

• M. A. Fischler, R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Comm. of the ACM, Vol 24, pp 381-395, 1981. 28

RANSAC: random sample consensus RANSAC loop (for N iterations): • Select four feature pairs (at random) • Compute homography H • Count inliers where || p i ’ - H p i || < ε Afterwards: • Choose largest set of inliers • Recompute H using only those inliers (often using high-quality nonlinear least squares) 29 Source: Torralba, Freeman, Isola

Simple example: fit a line • Rather than homography H (8 numbers) 
 fit y=ax+b (2 numbers a, b) to 2D pairs 30 Source: Torralba, Freeman, Isola

Simple example: fit a line • Pick 2 points • Fit line • Count inliers 3 inlier 31 Source: Torralba, Freeman, Isola

Simple example: fit a line • Pick 2 points • Fit line • Count inliers 4 inlier 32 Source: Torralba, Freeman, Isola

Simple example: fit a line • Pick 2 points • Fit line • Count inliers 9 inlier 33 Source: Torralba, Freeman, Isola

Simple example: fit a line • Pick 2 points • Fit line • Count inliers 8 inlier 34 Source: Torralba, Freeman, Isola

Simple example: fit a line • Use biggest set of inliers • Do least-square fit 35 Source: Torralba, Freeman, Isola

Warping with a homography 1. Compute features using SIFT 2. Match features 3. Compute homography using RANSAC 36 Source: N. Snavely

Estimating 3D structure • Given many images, how can we a) figure out where they were all taken from? b) build a 3D model of the scene? This is the structure from motion problem 37 Source: N. Snavely

Structure from motion Reconstruction (side) (top) • Input: images with points in correspondence p i , j = ( u i , j , v i , j ) • Output • structure: 3D location x i for each point p i • motion: camera parameters R j , t j possibly K j • Objective function: minimize reprojection error 38 Source: N. Snavely

Camera calibration & triangulation • Suppose we know 3D points – And have matches between these points and an image – Computing camera parameters similar to homography estimation • Suppose we have know camera parameters, each of which observes a point – How can we compute the 3D location of that point? • Seems like a chicken-and-egg problem, but in SfM we can solve both at once 39 Source: N. Snavely

Feature detection Detect features using SIFT [Lowe, IJCV 2004] 40 Source: N. Snavely

Feature detection Detect features using SIFT [Lowe, IJCV 2004] 41 Source: N. Snavely

Feature matching Match features between each pair of images 42 Source: N. Snavely

Feature matching Refine matching using RANSAC to estimate fundamental matrix between each pair 43 Source: N. Snavely

Correspondence estimation • Link up pairwise matches to form connected components of matches across several images Image 1 Image 2 Image 3 Image 4 44 Source: N. Snavely

Image connectivity graph 45 Source: N. Snavely

Structure from motion X 4 X 1 X 3 minimize g( R , T , X ) X 2 non-linear least squares X 5 X 7 X 6 p 1,1 p 1,3 p 1,2 Camera 1 Camera 3 R 1 ,t 1 R 3 ,t 3 Camera 2 R 2 ,t 2 46 Source: N. Snavely

Structure from motion • Minimize sum of squared reprojection errors: predicted observed image location image location indicator variable : is point i visible in image j ? • Minimizing this function is called bundle adjustment – Optimized using non-linear least squares, e.g. Levenberg-Marquardt 47 Source: N. Snavely

48

49

50

Multi-view stereo We have the camera pose. Estimate depth using stereo! Source: N. Snavely

Source: N. Snavely

Source: N. Snavely