C18 Computer Vision Lecture 7 Recovering 3D from two images I: epipolar geometry Victor Adrian Prisacariu http://www.robots.ox.ac.uk/~victor
Computer Vision: This time… 5. Imaging geometry, camera calibration. 6. Salient feature detection and description. 7. 7. Recoveri ring 3D from two im images I: I: epip ipola lar geometry ry. 1. Introduction 2. Epipolar Geometry 3. Algebraic Representation and the Fundamental Matrix 4. Computing the Fundamental Matrix 5. The Essential Matrix and Ego-Motion 8. Recovering 3D from two images II: stereo correspondences, triangulation, neural nets.
7.1 Introduction: Forward and Inverse Mappings • Geometric image entities back-proje ject to entities of higher dimensionality in the scene. • So points in the image back-project to lin lines in the scene, lines to planes, etc.
Introduction: What do single-view ambiguities tell us? • Sing ngle le vie views ar are e NOT T suf suffic icie ient to solve geometric problem in data- driven vision. • We need methods of understanding mult ultip iple vie views. – Shape-from-stereo: different cameras, different viewpoints, same time. – Structure-from-motion: same camera, different viewpoints, different times. • Note that single views may be sufficient to solve geometric problems in model (easy) or data driven vision (just recently) vision.
Reconstruction from monocular views
Introduction: Reconstruction from two views In principle, recovering 3D structure is straightforward: Fin Find a a bit bit of of the the tw two o sce scenes tha that is s ob observable by tw two o or or mor ore e cameras, and and bac backproject the the tw two o rays to o fi find the their ir intersectio ion in n the the worl orld. There are three things to cover: 1. Understanding the geometry – epip ipolar geometry ry. 2. Determining which points in the images are from the same scene location – th the e cor orresponden ence problem. 3. Determining the 3D structure by back-projecting rays – tri triangu gulation.
Simple case: two identical parallel cameras • Assume: – Some 3D world. – Two views. 3D – Two identical cameras – same camera World params (f). – Separation only on x: 𝑢 𝑦 • 𝑍 𝐷 1 = 𝑍 𝐷 2 , 𝑎 𝐷 1 = 𝑎 𝐷 2 , 𝑌 𝐷 1 = 𝑌 𝐷 2 + 𝑢 𝑦 View 1 View 2 • 𝑦 𝐷 1 = 𝑔𝑌 𝐷 1 /𝑎 and 𝑦 𝐷 2 = 𝑔𝑌 𝐷 2 /𝑎 so 𝑎 = 1 1 (𝑦 𝐷 1 − 𝑦 𝐷 2 ) 𝑔𝑢 𝑦 𝐷 1 𝐷 2 • Recip eciprocal dep epth th 1/Z is proportional to isparity 𝑦 𝐷 1 − 𝑦 𝐷 2 . the hori orizontal l dis 𝑢 𝑦
Simple case: two identical parallel cameras 𝑦 𝐷 2 = 𝑔 𝑦 𝐷 2 = 𝑦 + 𝑔𝑢 𝑦 𝑎 (𝑌 + 𝑢 𝑦 ) 𝑎 ൞ ⇒ 𝑧 𝐷 2 = 𝑔 𝑧 𝐷 2 = 𝑔 3D 𝑎 𝑍 𝑎 𝑍 World 𝑎 ranges from 0 to ∞ View 1 View 2 𝐷 1 𝐷 2 All possible matches lie on a straight lin ine. Finding a correspondence is a 1D 1D sear search. 𝑢 𝑦
7.2 Arbitrary case: Epipolar Geometry • The locus of the matches is the projection onto one view of the backprojected ray in the other one. • This is al always a a str traig ight line, and is called the ep epip ipola lar line, which we label with 𝐦’ . • As 𝐘 moves along the ray, the other ray sweeps across the ep epip ipolar pla plane. • The intersection of the epipolar plane with the image plane is the epipolar line.
Arbitrary case: Epipolar Geometry • If 𝐘 is moved, the entire plane moves too, generating a new epipolar line. • Epipolar planes hinge about the camera bas baseli line, forming a pencil of planes. • le 𝐟 ’ , where the baseline All epipolar lines in C’ meet at its ep epip ipole pierces the image plane. • The epipole point is the pr proje ojectio ion of of the the op optic ical l cen center of camera C into C’.
Epipolar geometry examples 1. Converging cameras Notice that the epipoles must often lie off the physical image planes. What would be a quick test you could carry out to see whether the epipole was on the image plane?
Epipolar geometry examples 2. (Close to) parallel cameras Epipolar geometry depends only on the relative post of the cameras (i.e. the rotation and translation between them) and on the camera’s intrinsic parameters. It It does not t depend on th the sc scene str truct ctrure. Can you reason qualitatively why not?
7.3 Algebraic representation and the F matrix Before formulating the algebraic representation we must cover three things: 1. How homogeneous notation handles points at infinity. 2. How homogeneous notation can be generalised to lines. You must notice a duality between lines and points … 3. How vector product can be represented using matrices.
1> Points at ∞ in homogeneous coordinates • A line of points in 3D through the point 𝐁 with direction 𝐄 is: 𝐘 = 𝐁 + μ𝐄 • In homogeneous notation this becomes: 𝑌 1 (𝜈) 0 = 1 𝑌 2 (𝜈) = 𝐁 + 𝜈𝐄 = 𝐁 1 + 𝜈 𝐄 𝐁 1 + 𝐄 0 . 𝑌 3 (𝜈) 1 𝜈 𝑌 4 (𝜈) As 𝜈 → ∞ , the point on the line becomes 𝐄 • 0 . • So homogeneous vectors with 𝒀 𝟓 =0 represent points at “infinity” . – Points at infinity are equivalent to directions. – Parallel lines in the scene meet at the same point.
1> Points at ∞ in homogeneous coordinates • The projection of a point at ∞ into the image is its vanishing poin int. • To find it, simply project the point-at- ∞ into the image: 𝐄 𝐰 = 𝑳 𝑺 𝐮 0
1> Points at ∞ in homogeneous coordinates
2> Homogeneous notation for lines • In 2D, a point 𝑦, 𝑧 T is represented in homogeneous coordinates as 𝐲 = 𝑦 1 , 𝑦 2 , 𝑦 3 T . • where 𝑦 = 𝑦 1 /𝑦 3 and 𝑧 = 𝑧 1 /𝑧 3 . • The lin line l 1 𝑦 + l 2 y + l 3 = 0 in 2D is represented by the homogeneous 3-vector: 𝐦 = 𝑚 1 , 𝑚 2 , 𝑚 3 T • Any point 𝐲 on the line 𝐦 has: 𝐉 T 𝐲 = 𝐲 T 𝐉 = 0
2> Homogeneous notation for lines Reminders: • A point 𝐲 on the line 𝐦 has 𝐦 T 𝐲 = 0 , 𝐲 T 𝐦 = 0 and 𝐦 ∙ 𝐲 = 0 . • The line through two points 𝐪 and 𝐫 is given by: 𝐦 = 𝐪 × 𝐫 • The intersection of two lines is the point: 𝐲 = 𝐦 1 × 𝐦 2
Ƹ Ƹ 3> Matrix representation of vector products The vector product 𝐛 × 𝐜 is: መ 𝑏 2 𝑐 3 − 𝑏 3 𝑐 2 0 −𝑏 3 𝑏 2 𝑐 1 𝐣 𝐤 𝐥 𝑏 3 𝑐 1 − 𝑏 1 𝑐 3 𝑏 3 0 −𝑏 1 𝑐 2 = = = 𝒃 𝑦 𝐜 𝑏 1 𝑏 2 𝑏 3 −𝑏 2 𝑏 1 0 𝑏 1 𝑐 2 − 𝑏 2 𝑐 1 𝑐 3 𝑐 1 𝑐 2 𝑐 3 𝒃 𝒚 is a 3 × 3 skew-symmetric matrix and has rank=2 𝒃 is the kernel of 𝒃 𝒚 𝐜 Example: compute the vector product of I=(1,2,3) and m=(2,3,4). Pseudo-determinant method gives (-1, 2, 1) T Skew-sym method gives: 0 −3 2 2 = 3 0 −1 3 −2 1 0 4
Algebraic representation of Epipolar Geometry • We now know that epipolar geometry defines the mapping fr from poin int t 𝐲 to lin line 𝐦’ . • The mapping depends only on the cameras, not on the structure, so the mapping depends on th the overall proje jection matr trices P and P’ . • We will next show that this mapping is linear, and can be written as: 𝑱’ = 𝑮𝒚 , , where 𝑮 is is th the fu fundamental matr trix
Algebraic representation of Epipolar Geometry • We use the first camera C to define the world coordinate system, so the overall projection matrix is: 𝑸 = 𝑳[𝑱|𝟏] • We then define the rotation and translation between camera frames as: 𝑺 𝐮 𝐘 ′ = 1 𝐘 𝟏 T • So that: 𝑸′ = 𝑳 ′ [𝑺|𝐮] • Note that the camera intrinsics 𝑳 and 𝑳’ can be different.
Algebraic representation of Epipolar Geometry • Step ep 1: back project a ray from C. • The back-projected 𝐘(𝜂) satisfies 𝐲 = 𝑳 𝑱 𝟏 𝐘 𝜂 • where we use 𝜂 as a parameter. • Now: X 𝑦 𝑌 𝑍 𝑧 𝑦 = = 𝑳 𝑱 𝟏 = 𝑳 ⇒ 𝑍 𝑎 1 𝑎 1 ⇒ 𝐘 𝜂 = 𝜂[𝑳] −1 𝐲 and 𝐘 ∞ = 𝑳 −1 𝐲 1 0 In effect, 𝑳 −1 corrects the direction of the ray. Direction 𝐲 o we be incorrect, because 𝐲 was measured in a non-ideal camera.
Algebraic representation of Epipolar Geometry • Step ep 2: Chose two points on the ray and project into second camera C’. – Point 1 with 𝜂 = 0 is the optical center 𝟏 1 . – Point 2 with 𝜂 = ∞ is the point at infinity 𝑳 −1 . 0 • Projecting into C’ we get: • Point 1: 𝟏 𝐟 ′ = 𝑳 ′ 𝑺 𝐮 1 = 𝑳 ′ 𝐮 . • Point 2: 𝑳 −𝟐 𝐫 = 𝑳 ′ 𝑺 𝐮 = 𝑳 ′ 𝑺𝑳 −1 𝐲 . 0
Algebraic representation of Epipolar Geometry • Step 3: Use vector product to find the epipolar line 𝐦 ′ = 𝑳 ′ 𝐮 × 𝑳 ′ 𝑺𝑳 −1 𝐲 • Next we tidy up: – Use 𝑵𝐛 × 𝑵𝐜 = 𝑵 −T (𝐛 × 𝐜) , where 𝑵 −T = 𝑵 −1 T – Use the matrix representation of a vector product i.e. the matrix 𝐮 𝑦 • So 𝐦 ′ = 𝑳 ′ −T 𝐮 × 𝑺𝑳 −𝟐 𝐲 = 𝑳 ′ −𝐔 𝐮 𝐲 𝑺𝑳 −1 𝐲 • 𝑮 = 𝑳 ′ −𝐔 𝐮 𝐲 𝑺𝑳 −1 is known as the fu fundamental l matrix, with the epipole line 𝐦′ = 𝑮𝐲 and the property that 𝐲 ′ 𝑮𝐲 = 0
Recommend
More recommend
