3D Vision Marc Pollefeys Viktor Larsson Spring 2020 Schedule Feb - PowerPoint PPT Presentation

3D Vision Marc Pollefeys Viktor Larsson Spring 2020

Schedule Feb 17 Introduction Feb 24 Geometry, Camera Model, Calibration Mar 2 Features, Tracking / Matching Mar 9 Project Proposals by Students Mar 16 Structure from Motion (SfM) + papers Mar 23 Dense Correspondence (stereo / optical flow) + papers Mar 30 Bundle Adjustment & SLAM + papers Apr 6 Student Midterm Presentations Apr 13 Easter break Multi-View Stereo & Volumetric Modeling + papers Apr 20 Apr 27 3D Modeling with Depth Sensors + papers May 4 3D Scene Understanding + papers May 11 4D Video & Dynamic Scenes + papers May 18 papers May 25 Student Project Demo Day = Final Presentations

3D Vision – Class 2 Projective Geometry and Camera Model points, lines, planes, conics and quadrics Transformations, camera model Read tutorial chapter 2 and 3.1 http://www.cs.unc.edu/~marc/tutorial/ Chapters 1, 2 and 5 in Hartley and Zisserman 1 st edition Or Chapters 2, 3 and 6 in 2 nd edition See also Chapter 2 in Szeliski book

Topics Today • Lecture intended as a review of material covered in Computer Vision lecture • Probably the hardest lecture (since very theoretic) in the class … • … but fundamental for any type of 3D Vision application • Key takeaways: • 2D primitives (points, lines, conics) and their transformations • 3D primitives and their transformations • Camera model and camera calibration

Overview • 2D Projective Geometry • 3D Projective Geometry • Camera Models & Calibration

2D Projective Geometry? Projections of planar surfaces A. Criminisi. Accurate Visual Metrology from Single and Multiple Uncalibrated Images . PhD Thesis 1999.

2D Projective Geometry? Measure distances A. Criminisi. Accurate Visual Metrology from Single and Multiple Uncalibrated Images . PhD Thesis 1999. reflected fix defined. find filter

2D Projective Geometry? Discovering details Piero della Francesca, La Flagellazione di Cristo (1460) A. Criminisi. Accurate Visual Metrology from Single and Multiple Uncalibrated Images . PhD Thesis 1999. Rectification floor rectified rectified rectified floor Rectification floor rectified figure rectified rectified floor figure

2D Projective Geometry? Image Stitching

2D Projective Geometry? Image Stitching

2D Euclidean Transformations • Rotation (around origin) 𝑦′′ 𝑦 𝑧′′ 𝑦′ cos 𝛽 − sin 𝛽 𝑧′ = 𝑦′ 𝑧 sin 𝛽 cos 𝛽 𝑧′ 𝑦 • Translation 𝛽 𝑧 𝑧′ + 𝑢 𝑦 𝑦′′ 𝑦′ = 𝑧 ′′ 𝑢 𝑧 • “Extended coordinates” cos 𝛽 − sin 𝛽 𝑢 𝑦 𝑦 𝑦′′ sin 𝛽 cos 𝛽 𝑢 𝑧 𝑧 = 𝑧′′ 1 1 0 0 1

Homogeneous Coordinates Homogenous coordinates y z Equivalence class of vectors z=1 −9 3 6 = −4 6 −2 = x 1 2 −3 ℙ 2 = ℝ 3 \ { 0,0,0 } 2D projective space:

Homogeneous Coordinates l (Homogeneous) representation of 2D line: c / a 2 + b 2 T x,y, 1 ( ) = 0 ( ) ax + by + c = 0 a,b,c l T x = 0 The point x lies on the line l if and only if Note that scale is unimportant for incidence relation T ~ k x , y ,1 ( ) ( ) T ~ k a,b,c ( ) ( ) T , " k ¹ 0 T , " k ¹ 0 x , y ,1 a,b,c ( ) T x 1 , x 2 , x 3 Homogeneous coordinates but only 2DOF T = x 1 x 3 , x 2 x 3 ( ) ( ) T Inhomogeneous coordinates x , y

2D Projective Transformations Definition: A projectivity is an invertible mapping h from ℙ 2 to itself such that three points x 1 , x 2 , x 3 lie on the same line if and only if h (x 1 ), h (x 2 ), h (x 3 ) do. Theorem: A mapping h : ℙ 2 → ℙ 2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P 2 represented by a vector x it is true that h (x)= H x Definition: Projective transformation       x ' 1 h 11 h 12 h 13 x 1       = x' = H x x ' 2    h 21 h 22 h 23   x 2  or 8DOF             x ' 3 h 31 h 32 h 33 x 3 projectivity = collineation = proj. transformation = homography ฀ ฀

Hierarchy of 2D Transformations transformed invariants squares   h h h 11 12 13 Concurrency, collinearity,   Projective order of contact (intersection, h h h   21 22 23 tangency, inflection, etc.), 8dof     cross ratio h h h 31 32 33 Parallelism, ratio of areas,   a a t ratio of lengths on parallel 11 12 x   Affine lines (e.g. midpoints), linear a a t   6dof 21 22 y combinations of vectors     (centroids), 0 0 1 The line at infinity l ∞   sr sr t 11 12 x   Ratios of lengths, angles, Similarity sr sr t The circular points I,J   21 22 y 4dof     0 0 1   r r t 11 12 x   Euclidean Absolute lengths, angles, r r t   3dof 21 22 y areas    0 0 1 

Working with Homogeneous Coordinates 𝑦 H 𝑧 𝑦′ 𝑧′ Type equation here. • “Homogenize”: • Apply H : • De-homogenize:

Lines to Points, Points to Lines 𝑚 2 𝑚 1 • Intersections of lines 𝑦 𝑈 𝑦 = 0 ൝ 𝑚 1 𝑦 = 𝑚 1 × 𝑚 2 𝑦 Find such that 𝑈 𝑦 = 0 𝑚 2 • Line through two points 𝑚 𝑦 1 𝑦 2 ൝𝑚 𝑈 𝑦 1 = 0 𝑚 𝑚 = 𝑦 1 × 𝑦 2 Find such that 𝑚 𝑈 𝑦 2 = 0

Transformation of Points and Lines 𝐼 • For a point transformation 𝑦 ′ = 𝐼𝑦 • Transformation for lines 𝐼 −𝑈 𝑚 ′ = 𝐼 −𝑈 𝑚 𝑚 𝑈 𝑦 = 0 (𝐼 −𝑈 𝑚) 𝑈 𝐼𝑦 = 0 𝑚 𝑈 (𝐼 −1 𝐼)𝑦 = 0 𝑚 ′ 𝑦′

Ideal Points • Intersections of parallel lines? 𝑏 𝑏 𝑐 𝑐 𝑐 𝑚 1 × 𝑚 2 = × = (𝑑′ − 𝑑) −𝑏 𝑑 𝑑′ 0 𝑚 1 = (𝑏, 𝑐, 𝑑) 𝑚 2 = (𝑏, 𝑐, 𝑑′) ( ) T • Parallel lines intersect in Ideal Points x 1 , x 2 ,0

Ideal Points • Ideal points correspond to directions 𝑐 𝑚 1 = (𝑏, 𝑐, 𝑑) Ideal point −𝑏 0 (𝑏, 𝑐) (𝑐, −𝑏) • Unaffected by translation 𝑠 𝑠 𝑢 𝑦 𝑦 𝑠 11 𝑦 + 𝑠 12 𝑧 11 12 𝑠 𝑠 𝑢 𝑧 𝑧 = 𝑠 21 𝑦 + 𝑠 22 𝑧 21 22 0 0 0 0 1

The Line at Infinity • Line through two ideal points? 𝑦 0 𝑦′ 0 𝑧 0 × = = = 𝑚 ∞ 0 𝑧′ 𝑦𝑧 ′ − 𝑦 ′ 𝑧 0 1 0 ( ) l = • Line at infinity intersects all ideal points T 0 , 0 , 1  𝑦 1 𝑈 𝑦 = 𝑚 ∞ 𝑈 𝑦 2 𝑚 ∞ = 𝑦 3 = 0 𝑦 3 Note that in ℙ 2 there is no distinction ℙ 2 = ℝ 2 ∪ 𝑚 ∞ between ideal points and others

The Line at Infinity The line at infinity l  =(0,0,1) T is a fixed line under a projective transformation H if and only if H is an affinity (affine transformation)   0    − T A 0  = − = = T    l H l 0 l    − − A T T    t A 1   1 Affine trans. Note: not fixed pointwise 𝑩 𝒖 𝑰 𝐵 = 𝟏 𝑼 1

Conics • Curve described by 2 nd -degree equation in the plane Parabola Ellipse Hyperbola Circle Image source: Wikipedia

Conics • Curve described by 2 nd -degree equation in the plane 𝑏𝑦 2 + 𝑐𝑦𝑧 + 𝑑𝑧 2 + 𝑒𝑦 + 𝑓𝑧 + 𝑔 = 0 or homogenized 2 + 𝑐𝑦 1 𝑦 2 + 𝑑𝑦 2 2 + 𝑒𝑦 1 𝑦 3 + 𝑓𝑦 2 𝑦 3 + 𝑔𝑦 3 2 = 0 𝑏𝑦 1 or in matrix form 𝒚 𝑈 𝐷𝒚 = 0 𝑦 1 𝑏 𝑐/2 𝑒/2 𝑦 2 𝑐/2 𝑑 𝑓/2 𝑦 1 𝑦 2 𝑦 3 = 0 𝑦 3 𝑒/2 𝑓/2 𝑔 { } a : b : c : d : e : f • 5DOF (degrees of freedom): (defined up to scale)

Five Points Define a Conic For each point the conic passes through 2 + bx i y i + cy i 2 + dx i + ey i + f = 0 ax i or ( ) ( ) T c = a , b , c , d , e , f = 2 2 x , x y , y , x , y , 1 c 0 i i i i i i stacking constraints yields   2 2 x 1 x 1 y 1 y 1 x 1 y 1 1   2 2   x 2 x 2 y 2 y 2 x 2 y 2 1   c = 0 2 2 x 3 x 3 y 3 y 3 x 3 y 3 1   2 2  x 4 x 4 y 4 y 4 x 4 y 4 1     2 2  x 5 x 5 y 5 y 5 x 5 y 5 1 ฀

Tangent Lines to Conics The line l tangent to C at point x on C is given by l = Cx l x C

Dual Conics * = T • A line tangent to the conic C satisfies l C l 0 C * = C -1 • In general ( C full rank): • Dual conics = line conics = conic envelopes ฀

Degenerate Conics • A conic is degenerate if matrix C is not of full rank m e.g. two lines (rank 2) l C = lm T + ml T ฀ e.g. repeated line (rank 1) ฀ C = ll T ฀ l • Degenerate line conics: 2 points (rank 2), double point (rank1) ฀ *  C ฀ ( ) C * • Note that for degenerate conics ฀

Transformation of Points, Lines and Conics • For a point transformation 𝑦 ′ = 𝐼𝑦 • Transformation for lines 𝑚 ′ = 𝐼 −𝑈 𝑚 • Transformation for conics 𝐷 ′ = 𝐼 −𝑈 𝐷𝐼 −1 • Transformation for dual conics 𝐷 ∗′ = 𝐼𝐷 ∗ 𝐼 𝑈

Application: Removing Perspective Two stages: • From perspective to affine transformation via the line at infinitiy • From affine to similarity transformation via the circular points