Camera Calibration and Stereo Various slides from previous courses - PowerPoint PPT Presentation

CS4501: Introduction to Computer Vision Camera Calibration and Stereo Various slides from previous courses by: D.A. Forsyth (Berkeley / UIUC), I. Kokkinos (Ecole Centrale / UCL). S. Lazebnik (UNC / UIUC), S. Seitz (MSR / Facebook), J. Hays (Brown / Georgia Tech), A. Berg (Stony Brook / UNC), D. Samaras (Stony Brook) . J. M. Frahm (UNC), V. Ordonez (UVA).

Last Class • Line Detection using the Hough Transform • Least Squares / Hough Transform / RANSAC

Today’s Class • Camera Calibration • Stereo Vision

Hough transform • An early type of voting scheme • General outline: • Discretize parameter space into bins • For each feature point in the image, put a vote in every bin in the parameter space that could have generated this point • Find bins that have the most votes Image space Hough parameter space P.V.C. Hough, Machine Analysis of Bubble Chamber Pictures, Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959 Slide by Svetlana Lazebnik

Algorithm outline • Initialize accumulator H to all zeros • For each feature point (x,y) in the image ρ For θ = 0 to 180 ρ = x cos θ + y sin θ H(θ, ρ) = H(θ, ρ) + 1 θ end end • Find the value(s) of (θ, ρ) where H(θ, ρ) is a local maximum • The detected line in the image is given by ρ = x cos θ + y sin θ Slide by Svetlana Lazebnik

Hough Transform Example

RANSAC (RANdom SAmple Consensus) : Fischler & Bolles in ‘81. Algorithm: 1. Sample (randomly) the number of points required to fit the model 2. Solve for model parameters using samples 3. Score by the fraction of inliers within a preset threshold of the model Repeat 1-3 until the best model is found with high confidence

RANSAC Line fitting example Algorithm: 1. Sample (randomly) the number of points required to fit the model (#=2) 2. Solve for model parameters using samples 3. Score by the fraction of inliers within a preset threshold of the model Repeat 1-3 until the best model is found with high confidence Illustration by Savarese

RANSAC Line fitting example Algorithm: 1. Sample (randomly) the number of points required to fit the model (#=2) 2. Solve for model parameters using samples 3. Score by the fraction of inliers within a preset threshold of the model Repeat 1-3 until the best model is found with high confidence

RANSAC Line fitting example d = N 6 I Algorithm: 1. Sample (randomly) the number of points required to fit the model (#=2) 2. Solve for model parameters using samples 3. Score by the fraction of inliers within a preset threshold of the model Repeat 1-3 until the best model is found with high confidence

RANSAC d = N 14 I Algorithm: 1. Sample (randomly) the number of points required to fit the model (#=2) 2. Solve for model parameters using samples 3. Score by the fraction of inliers within a preset threshold of the model Repeat 1-3 until the best model is found with high confidence

Camera Calibration • What does it mean?

Slide Credit: Silvio Saverese Recall the Projection matrix R,T j w k w O w i w [ ] X x = x : Image Coordinates: (u,v,1) K R t K : Intrinsic Matrix (3x3) R : Rotation (3x3) t : Translation (3x1) X : World Coordinates: (X,Y,Z,1)

Recall the Projection matrix [ ] X x = K R t ! = #[% '] =

Recall the Projection matrix [ ] X x = K R t ! = Goal: Find ) #[% '] =

Camera Calibration [ ] X x = K R t ! = # =

Camera Calibration [ ] X x = K R t ! = # = Goal: Find $[& (]

Slide Credit: James Hays Calibrating the Camera Use an scene with known geometry • Correspond image points to 3d points • Get least squares solution (or non-linear solution) Known 3d Known 2d locations image coords é ù X é ù é ù su m m m m ê ú 11 12 13 14 Y ê ú ê ú ê ú = sv m m m m ê ú ê ú 21 22 23 24 ê ú Z ê ú ê ú s m m m m ê ú ë û ë û 31 32 33 34 1 ë û Unknown Camera Parameters

How do we calibrate a camera? Slide Credit: James Hays Known 2d Known 3d image coords locations 880 214 312.747 309.140 30.086 43 203 305.796 311.649 30.356 270 197 307.694 312.358 30.418 886 347 310.149 307.186 29.298 745 302 311.937 310.105 29.216 943 128 311.202 307.572 30.682 476 590 307.106 306.876 28.660 419 214 309.317 312.490 30.230 317 335 307.435 310.151 29.318 783 521 308.253 306.300 28.881 235 427 306.650 309.301 28.905 665 429 308.069 306.831 29.189 655 362 309.671 308.834 29.029 427 333 308.255 309.955 29.267 412 415 307.546 308.613 28.963 746 351 311.036 309.206 28.913 434 415 307.518 308.175 29.069 525 234 309.950 311.262 29.990 716 308 312.160 310.772 29.080 602 187 311.988 312.709 30.514

Unknown Camera Parameters Slide Credit: James Hays é X ù é su ù é m m m m ù ê ú 11 12 13 14 Known 2d Known 3d Y ê ú ê ú ê ú = sv m m m m ê ú ê ú image coords locations 21 22 23 24 ê ú Z ê ú ê ú s m m m m ê ú ë û ë û 31 32 33 34 1 ë û = + + + su m X m Y m Z m 11 12 13 14 = + + + sv m X m Y m Z m 21 22 23 24 = + + + s m X m Y m Z m 31 32 33 34 + + + m X m Y m Z m = u 11 12 13 14 + + + m X m Y m Z m 31 32 33 34 + + + m X m Y m Z m = v 21 22 23 24 + + + m X m Y m Z m 31 32 33 34

Unknown Camera Parameters Slide Credit: James Hays é X ù é su ù é m m m m ù ê ú 11 12 13 14 Known 2d Known 3d Y ê ú ê ú ê ú = sv m m m m ê ú ê ú image coords locations 21 22 23 24 ê ú Z ê ú ê ú s m m m m ê ú ë û ë û 31 32 33 34 1 ë û + + + m X m Y m Z m = u 11 12 13 14 + + + m X m Y m Z m 31 32 33 34 + + + m X m Y m Z m = v 21 22 23 24 + + + m X m Y m Z m 31 32 33 34 + + + = + + + ( m X m Y m Z m ) u m X m Y m Z m 31 32 33 34 11 12 13 14 + + + = + + + ( m X m Y m Z m ) v m X m Y m Z m 31 32 33 34 21 22 23 24 + + + = + + + m uX m uY m uZ m u m X m Y m Z m 31 32 33 34 11 12 13 14 + + + = + + + m vX m vY m vZ m v m X m Y m Z m 31 32 33 34 21 22 23 24

Unknown Camera Parameters Slide Credit: James Hays é X ù é su ù é m m m m ù ê ú 11 12 13 14 Known 2d Known 3d Y ê ú ê ú ê ú = sv m m m m ê ú ê ú image coords locations 21 22 23 24 ê ú Z ê ú ê ú s m m m m ê ú ë û ë û 31 32 33 34 1 ë û + + + = + + + m uX m uY m uZ m u m X m Y m Z m 31 32 33 34 11 12 13 14 + + + = + + + m vX m vY m vZ m v m X m Y m Z m 31 32 33 34 21 22 23 24 = + + + - - - - 0 m X m Y m Z m m uX m uY m uZ m u 11 12 13 14 31 32 33 34 = + + + - - - - 0 m X m Y m Z m m vX m vY m vZ m v 21 22 23 24 31 32 33 34

Unknown Camera Parameters Slide Credit: James Hays é X ù é su ù é m m m m ù ê ú 11 12 13 14 Y Known 2d ê ú ê ú Known 3d ê ú = sv m m m m ê ú ê ú image coords 21 22 23 24 ê ú locations Z ê ú ê ú s m m m m ë û ë û ê ú 31 32 33 34 1 ë û = + + + - - - - 0 m X m Y m Z m m uX m uY m uZ m u 11 12 13 14 31 32 33 34 = + + + - - - - 0 m X m Y m Z m m vX m vY m vZ m v 21 22 23 24 31 32 33 34 • Method 1 – homogeneous linear é m ù 11 ê ú system. Solve for m’s entries using m ê ú 12 ê ú m linear least squares 13 ê ú m ê ú - - - - 14 é X Y Z 1 0 0 0 0 u X u Y u Z u ù é 0 ù ê ú 1 1 1 1 1 1 1 1 1 1 m ê ú ê ú [U, S, V] = svd(A); - - - - 21 0 0 0 0 X Y Z 1 v X v Y v Z v ê ú 0 ê ú ê ú 1 1 1 1 1 1 1 1 1 1 m ê ú ê ú 22 = ê ú ! ! M = V(:,end); ê ú m ê ú ê ú - - - - ê 23 ú X Y Z 1 0 0 0 0 u X u Y u Z u 0 ê ú ê ú M = reshape(M,[],3)'; n n n n n n n n n n m ê ú ê ú ê ú - - - - 24 0 0 0 0 X Y Z 1 v X v Y v Z v 0 ë û ë û ê ú n n n n n n n n n n m ê ú 31 ê m ú 32 ê ú m ê ú 33 ê ú m ë û 34