3D Vision Torsten Sattler Spring 2018 Schedule Feb 19 - PowerPoint PPT Presentation

2D Projective Transformations Definition: A projectivity is an invertible mapping h from P 2 to itself such that three points x 1 , x 2 , x 3 lie on the same line if and only if h (x 1 ), h (x 2 ), h (x 3 ) do. Theorem: A mapping h : P 2  P 2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P 2 represented by a vector x it is true that h (x)= H x Definition: Projective transformation       x ' 1 h 11 h 12 h 13 x 1        x'  H x x ' 2 h 21 h 22 h 23 x 2       or 8DOF        x ' 3   h 31 h 32 h 33   x 3  projectivity = collineation = proj. transformation = homography

Hierarchy of 2D Transformations transformed invariants squares   h h h 11 12 13 Concurrency, collinearity,   Projective order of contact (intersection, h h h   21 22 23 tangency, inflection, etc.), 8dof   h h h cross ratio   31 32 33 Parallellism, ratio of areas,   a a t ratio of lengths on parallel 11 12 x   Affine lines (e.g. midpoints), linear a a t   6dof 21 22 y combinations of vectors   0 0 1 (centroids),   The line at infinity l ∞   sr sr t 11 12 x Ratios of lengths, angles,   Similarity sr sr t The circular points I,J   4dof 21 22 y   0 0 1     r r t 11 12 x   Euclidean Absolute lengths, angles, r r t   3dof 21 22 y areas   0 0 1  

Fixed Points and Lines H e = λ e (eigenvectors H = fixed points) ( l 1 = l 2  pointwise fixed line) H - T l  l l (eigenvectors H - T = fixed lines)

Application: Removing Perspective Two stages: • From perspective to affine transformation via the line at infinitiy • From affine to similarity transformation via the circular points

Affine Rectification projection affine metric rectification rectification   1 0 0   [ ]   T  ¹ H H 0 1 0 H l l l l , l 0   P A  P 1 2 3 3   l l l   1 2 3

Affine Rectification v 1 l ∞ v 2 l   v 1 ´ v 2 l 1 l 3 v 1  l 1 ´ l 2 v 2  l 3 ´ l 4 l 2 l 4   1 0 0   0 1 0     l l l   1 2 3

Metric Rectification • Need to measure a quantity that is not invariant under affine transformations

The Circular Points The circular points I, J are fixed points under the projective transformation H iff H is a similarity         s cos s sin t 1 1     x             i I H I s sin s cos t i se i I       S y       0 0 1 0 0      

The Circular Points • every circle intersects l ∞ at the “circular points” l ∞ 2 + x 2 2 + dx 1 x 3 + ex 2 x 3 + fx 3 2  0 2 + x 2 2  0 x 1 x 1 x 3  0 ( ) T I  1, i ,0 ( ) T J  1,- i ,0 • Algebraically, encodes orthogonal directions T + i 0,1,0 ( ) ( ) T I  1,0,0

Conic Dual to the Circular Points ∞  * * T   1 0 0 C H C H S S ∞    * C 0 1 0   ∞   0 0 0   * C The dual conic is fixed conic under the ∞ projective transformation H iff H is a similarity * C Note: has 4DOF (det = 0) ∞ l ∞ is the nullvector

Conic Dual to the Circular Points l ∞   1 0 0     + * * T T C IJ JI C 0 1 0 I   ∞ ∞   0 0 0   J ∞  T * * C H C H S S ∞ * C The dual conic is fixed conic under the ∞ projective transformation H iff H is a similarity * C Note: has 4DOF (det = 0) ∞ l ∞ is the nullvector

Measuring Angles via the Dual Conic ( ) ( ) T T l  l m  m 1 , m 2 , m 3 • Euclidean: 1 , l 2 , l 3 1 m 1 + l 2 m 2 l cos   2 + l 2 2 + m 2 ( ) m 1 ( ) 2 2 l 1 l T C  * m cos   • Projective: ( ) m T C  ( ) l T C  * l * m l T C  * m  0 (orthogonal) • Knowing the dual conic on the projective plane, we can measure Euclidean angles!

Metric Rectification A -1 • Dual conic under affinity         *   A t I 0 A T  AA T 0 0    C      0 T 0 T t T 0 T  1   0   1   0  • S=AA T symmetric, estimate from two pairs of l T C  * m  0 orthogonal lines (due to ) Note: Result defined up to similarity

Update to Euclidean Space • Metric space: Measure ratios of distances • Euclidean space: Measure absolute distances • Can we update metric to Euclidean space? • Not without additional information

Important Points so far … • Definition of 2D points and lines • Definition of homogeneous coordinates • Definition of projective space • Effect of transformations on points, lines, conics • Next: Analogous concepts in 3D

Overview • 2D Projective Geometry • 3D Projective Geometry • Camera Models & Calibration

3D Points and Planes • 2D: duality point - line, 3D: duality point - plane • Homogeneous representation of 3D points and planes π X + π X + π X + π X = 0 1 1 2 2 3 3 4 4 • The point X lies on the plane π if and only if π T X = 0 • The plane π goes through the point X if and only if π T X = 0

Planes from Points T T T π X π = 0, X π = 0 X π = 0 Solve from and 1 2 3   T X   T X 1 1      T π X π 0 (solve as right nullspace of ) T X 2     2     T T X 3 X     3

Points from Planes T T T X π X = 0, π X = 0 π X = 0 Solve from and 1 2 3     T T π π 1 1      T X T π X 0 (solve as right nullspace of ) π     2 2     T T π π     3 3 Representing a plane by its span [ ]  R 4 ´ 3 M  X 1 X 2 X 3 X = M x T π M = 0

Lines in 3D • Representing a line by its span   T A  W λA +   μB T B   • Dual representation (4dof)   T P  W * λP + μQ   T Q   T T * * W W = WW = 0 2 × 2 • Example: X -axis     0 0 0 1 0 0 1 0   W * W     1 0 0 0 0 1 0 0     (Alternative: Plücker representation, details see e.g. H&Z)

Points, Lines and Planes W   W  M   M π = 0 X T X   W   W * X  M 0  M   π T π  

Quadrics and Dual Quadrics X T QX  0 ( Q : 4x4 symmetric matrix) • 9 DOF (up to scale) • In general, 9 points define quadric • det( Q )=0 ↔ degenerate quadric p  QX • tangent plane p T Q * p  0 Q * • Dual quartic: ( adjoint) Q *  Q -1 • relation to quadric (non-degenerate)

Transformation of 3D points, planes • Transformation for points (2D equivalent) X'  H X ( ) x'  H x • Transformation for planes ( ) l'  H -T l p '  H -T p

The Plane at Infinity The plane at infinity π  =(0, 0, 0, 1) T is a fixed plane under a projective transformation H iff H is an affinity   0       T 0 A 0      T π H π π       A  T T 0 -t A 1    1   ( ) T p   0,0,0,1 1. canonical position ( ) T D  X 1 , X 2 , X 3 ,0 2. contains all directions two planes are parallel  line of intersection in π ∞ 3. line || line (or plane)  point of intersection in π ∞ 4. 5. 2D equivalent: line at infinity

Transformation of 3D points, planes and quadrics • Transformation for points (2D equivalent) X'  H X ( ) x'  H x • Transformation for planes ( ) l'  H -T l p '  H -T p • Transformation for quadrics ( ) Q'  H -T QH -1 C '  H -T CH -1 • Transformation for dual quadrics Q' *  HQ * H T C ' *  HC * H T ( )

Hierarchy of 3D Transformations Projective Intersection and tangency 15dof Parallellism of planes, Volume ratios, centroids, Affine The plane at infinity π ∞ 12dof Similarity Angles, ratios of length 7dof The absolute conic Ω∞   R t Euclidean Volume   6dof T 0 1  

Hierarchy of 3D Transformations projective Plane at infinity affine Absolute conic similarity

The Absolute Conic • The absolute conic Ω ∞ is a (point) conic on π  • In a metric frame: ( ) I X 1 , X 2 , X 3 ( ) T or conic for directions: X 1 , X 2 , X 3 (with no real points) The absolute conic Ω ∞ is a fixed conic under the projective transformation H iff H is a similarity 1. Ω ∞ is only fixed as a set 2. Circles intersect Ω ∞ in two circular points 3. Spheres intersect π ∞ in Ω ∞

The Absolute Dual Quadric The absolute dual quadric Ω * ∞ is a fixed quadric under the projective transformation H iff H is a similarity 1. 8 dof 2. plane at infinity π ∞ is the nullvector of Ω ∞ 3. angles:

Important Points so far … • Def. of 2D points and lines, 3D points and planes • Def. of homogeneous coordinates • Def. of projective space (2D and 3D) • Effect of transformations on points, lines, planes • Next: Projections from 3D to 2D

Overview • 2D Projective Geometry • 3D Projective Geometry • Camera Models & Calibration

Camera Model Relation between pixels and rays in space ?

Pinhole Camera

Pinhole Camera Model

Pinhole Camera Model linear projection in homogeneous coordinates!

Pinhole Camera Model

Pinhole Camera Model x  PX [ ] P  diag ( f , f ,1) I |0

Principal Point Offset

Principal Point Offset ( p x , p y ) T principal point

Principal Point Offset [ ] X cam x  K I |0 calibration matrix

CCD Camera

Camera Rotation and Translation ˜ C

Camera Rotation and Translation ˜ C ( ) ˜ cam  R ˜ X - ˜ X C [ ] X [ ] X cam x  KR I |- ˜ x  K I |0 C [ ] t  - R ˜ x  PX ˜ P  K R | t C  - R T t C

General Projective Camera In practice:  x   y , s  0 ( ) = image p x , p y center [ ] P  KR I |- ˜ C 11 dof (5+3+3) non-singular [ ] P  K R | t intrinsic camera parameters extrinsic camera parameters

Affine Cameras

Camera Model Relation between pixels and rays in space ?

Projector Model Relation between pixels and rays in space (dual of camera) ? (main geometric difference is vertical principal point offset to reduce keystone effect)

Meydenbauer Camera vertical lens shift to allow direct ortho-photographs

Projections of Points and Lines x  PX • forward projection of point • image point back-projects to line / ray l X  R T K  1 x • forward projection of line (not point-wise!) ( )   +   +   +  X P(A B) PA ' PB a ' b • back-projection of line P  P T l ( ) l T x  0; x  PX P T X  l T PX

Projector-Camera Systems IR projector IR camera

Projections of Conics and Quadrics • back-projection of conic to cone x T Cx  X T P T CPX  0 Q co  P T CP ( ) x  PX • forward projection of quadric C *  PQ * P T P T Q * P  l T PQ * P T l  0 ( ) P  P T l

Image of Absolute Conic

Image of Absolute Conic w *  * P T P       R T I 0 [ ]  K T   K R t   0 T  0  t T    KK T K -T K  1 w 

A Simple Calibration Device (i) compute H for each square (corners @ (0,0),(1,0),(0,1),(1,1)) (ii) compute the imaged circular points H(1,±i,0) T (iii) Imaged circular points lie on w fit a conic to 6 circular points to obtain w (iv) compute K from w through Cholesky factorization (≈ Zhang’s calibration method)

Practical Camera Calibration Method and Pictures from Zhang (ICCV ’ 99): “ Flexible Camera Calibration By Viewing a Plane From Unknown Orientations ” Unknown: constant camera intrinsics K (varying) camera poses R,t Known: 3D coordinates of chessboard corners => Define to be the z=0 plane ( X=[X 1 X 2 0 1] T ) Point is mapped as λx = K (r 1 r 2 r 3 t) X λx = K (r 1 r 2 t) [X 1 X 2 1] ’ Homography H between image and chess coordinates, estimate from known X i and measured x i

Direct Linear Transformation (DLT)

Direct Linear Transformation (DLT) • Equations are linear in h : A i h  0 • Only 2 out of 3 are linearly independent (2 equations per point) (only drop third row if w i ’ ≠0) • Holds for any homogeneous representation, e.g. ( x i ’ , y i ’ ,1)

Direct Linear Transformation (DLT) • Solving for homography H Ah  0 size A is 8x9 (2eq.) or 12x9 (3eq.), but rank 8 T is not interesting • Trivial solution is h =0 9 • 1D null-space yields solution of interest pick for example the one with h  1

Direct Linear Transformation (DLT) • Over-determined solution Ah  0 • No exact solution because of inexact measurement, i.e., “ noise ” • Find approximate solution h  1 - Additional constraint needed to avoid 0 , e.g., Ah  0 Ah - not possible, so minimize

DLT Algorithm Objective Given n≥4 2D to 2D point correspondences { x i ↔ x i ’ }, determine the 2D homography matrix H such that x i ’ = Hx i Algorithm (i) For each correspondence x i ↔ x i ’ compute A i . Usually only two first rows needed. Assemble n 2x9 matrices A i into a single 2 n x9 matrix A (ii) (iii) Obtain SVD of A . Solution for h is last column of V (iv) Determine H from h

Importance of Normalization ~10 2 ~10 2 1 ~10 2 ~10 2 1 ~10 4 ~10 4 ~10 2 orders of magnitude difference! Monte Carlo simulation for identity computation based on 5 points (not normalized ↔ normalized)

Normalized DLT Algorithm Objective Given n≥4 2D to 2D point correspondences { x i ↔ x i ’ }, determine the 2D homography matrix H such that x i ’ = Hx i Algorithm (i) Normalize points (ii) Apply DLT algorithm to (iii) Denormalize solution Normalization (independently per image): • Translate points such that centroid is at origin 2 • Isotropic scaling such that mean distance to origin is

Normalized DLT Algorithm Objective Given n≥4 2D to 2D point correspondences { x i ↔ x i ’ }, determine the 2D homography matrix H such that x i ’ = Hx i Algorithm (i) Normalize points (ii) Apply DLT algorithm to (iii) Denormalize solution

Geometric Distance x measured coordinates ˆ x estimated coordinates x true coordinates d (.,.) Euclidean distance (in image) Error in one image e.g. calibration pattern Symmetric transfer error Reprojection error x' subject to

Reprojection Error

Statistical Cost Function and Maximum Likelihood Estimation • Optimal cost function related to noise model • Assume zero-mean isotropic Gaussian noise (assume outliers removed) ) ( ) 1 ( ) ( 2 2 e    d x, x / 2 Pr x 2 p 2 Error in one image Maximum Likelihood Estimate:  ( ) 2 min d x  i ,Hx i

Statistical Cost Function and Maximum Likelihood Estimation • Optimal cost function related to noise model • Assume zero-mean isotropic Gaussian noise (assume outliers removed) ) ( ) 1 ( ) ( 2 2 e    d x, x / 2 Pr x 2 p 2 Error in both images Maximum Likelihood Estimate  ( ) ( ) 2 2 + d d x i , ˆ i , ˆ min x x  x  i i