Back to the Homography: The Why Sanja Fidler CSC420: Intro to Image - PowerPoint PPT Presentation

Back to the Homography: The Why Sanja Fidler CSC420: Intro to Image Understanding 1 / 1

Homography In Lecture 9 we said that a homography is a transformation that maps a projective plane to another projective plane. We shamelessly dumped the following equation for homography without explanation: 2 3 2 3 2 3 x 0 a b c x 5 = y 0 w d e f y 4 4 5 4 5 1 1 g h i Sanja Fidler CSC420: Intro to Image Understanding 2 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: We have our object in two di ff erent worlds, in two di ff erent poses relative to camera, two di ff erent photographers, and two di ff erent cameras. Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Our object is a plane. Each plane is characterized by one point d on the plane and two independent vectors a and b on the plane. Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Then any other point X on the plane can be written as: X = d + α a + β b . Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Any two Chicken Run DVDs on our planet are related by some transformation T . We’ll compute it, don’t worry. Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Each object is seen by a di ff erent camera and thus projects to the corresponding image plane with di ff erent camera intrinsics. Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Given this, the question is what’s the transformation that maps the DVD on the first image to the DVD in the second image? Sanja Fidler CSC420: Intro to Image Understanding 3 / 1

Homography Each point on a plane can be written as: X = d + α · a + β · b , where d is a point, and a and b are two independent directions on the plane. Let’s have two di ff erent planes in 3D: First plane : X 1 = d 1 + α · a 1 + β · b 1 Second plane : X 2 = d 2 + α · a 2 + β · b 2 Via α and β , the two points X 1 and X 2 are in the same location relative to each plane. Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

Homography Each point on a plane can be written as: X = d + α · a + β · b , where d is a point, and a and b are two independent directions on the plane. Let’s have two di ff erent planes in 3D: First plane : X 1 = d 1 + α · a 1 + β · b 1 Second plane : X 2 = d 2 + α · a 2 + β · b 2 Via α and β , the two points X 1 and X 2 are in the same location relative to each plane. We can rewrite this using homogeneous coordinates: 2 3 2 3 α α ⇥ a 1 ⇤ 5 = A 1 First plane : X 1 = b 1 d 1 β β 4 4 5 1 1 2 3 2 3 α α ⇥ a 2 ⇤ 5 = A 2 Second plane : X 2 = b 2 d 2 β β 4 4 5 1 1 Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

Homography Each point on a plane can be written as: X = d + α · a + β · b , where d is a point, and a and b are two independent directions on the plane. Let’s have two di ff erent planes in 3D: First plane : X 1 = d 1 + α · a 1 + β · b 1 Second plane : X 2 = d 2 + α · a 2 + β · b 2 Via α and β , the two points X 1 and X 2 are in the same location relative to each plane. We can rewrite this using homogeneous coordinates: 2 3 2 3 α α ⇥ a 1 ⇤ 5 = A 1 First plane : X 1 = b 1 d 1 β β 4 4 5 1 1 2 3 2 3 α α ⇥ a 2 ⇤ 5 = A 2 Second plane : X 2 = b 2 d 2 β β 4 4 5 1 1 ⇥ a 1 ⇤ ⇥ a 2 ⇤ Careful: A 1 = and A 2 = are 3 × 3 matrices. b 1 d 1 b 2 d 2 Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

Homography Each point on a plane can be written as: X = d + α · a + β · b , where d is a point, and a and b are two independent directions on the plane. Let’s have two di ff erent planes in 3D: First plane : X 1 = d 1 + α · a 1 + β · b 1 Second plane : X 2 = d 2 + α · a 2 + β · b 2 Via α and β , the two points X 1 and X 2 are in the same location relative to each plane. We can rewrite this using homogeneous coordinates: 2 3 2 3 α α ⇥ a 1 ⇤ 5 = A 1 First plane : X 1 = b 1 d 1 β β 4 4 5 1 1 2 3 2 3 α α ⇥ a 2 ⇤ 5 = A 2 Second plane : X 2 = b 2 d 2 β β 4 4 5 1 1 ⇥ a 1 ⇤ ⇥ a 2 ⇤ Careful: A 1 = and A 2 = are 3 × 3 matrices. b 1 d 1 b 2 d 2 Sanja Fidler CSC420: Intro to Image Understanding 4 / 1

Homography In 3D, a transformation between the planes is given by: X 2 = T X 1 There is one transformation T between every pair of points X 1 and X 2 . Expand it: 2 3 2 3 α α 5 = T A 1 for every α , β A 2 β β 4 4 5 1 1 Then it follows: T = A 2 A − 1 1 , with T a 3 × 3 matrix. Let’s look at what happens in projective (image) plane. Note that we have each plane in a separate image and the two images may not have the same camera intrinsic parameters. Denote them with K 1 and K 2 . 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Sanja Fidler CSC420: Intro to Image Understanding 5 / 1

Homography From previous slide: 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Insert X 2 = T X 1 into equality on the right: 2 3 x 2 5 = K 2 T X 1 w 2 y 2 4 1 Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

Homography From previous slide: 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Insert X 2 = T X 1 into equality on the right: 2 3 x 2 5 = K 2 T X 1 = K 2 T ( K − 1 w 2 y 2 1 K 1 ) X 1 4 1 Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

Homography From previous slide: 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Insert X 2 = T X 1 into equality on the right: 2 3 x 2 5 = K 2 T X 1 = K 2 T ( K − 1 w 2 y 2 K 1 ) X 1 4 1 | {z } 1  x 1   y 1  w 1     1 Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

Homography From previous slide: 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Insert X 2 = T X 1 into equality on the right: 2 3 2 3 x 2 x 1 5 = K 2 T X 1 = K 2 T ( K − 1 1 K 1 ) X 1 = w 1 K 2 T K − 1 w 2 y 2 y 1 4 4 5 1 1 1 Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

Homography From previous slide: 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Insert X 2 = T X 1 into equality on the right: 2 3 2 3 x 2 x 1 5 = K 2 T X 1 = K 2 T ( K − 1 1 K 1 ) X 1 = w 1 K 2 T K − 1 w 2 y 2 y 1 4 4 5 1 | {z } 1 1 3 × 3 matrix Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

Homography From previous slide: 2 3 2 3 x 1 x 2 5 = K 1 X 1 5 = K 2 X 2 and w 1 y 1 w 2 y 2 4 4 1 1 Insert X 2 = T X 1 into equality on the right: 2 3 2 3 x 2 x 1 5 = K 2 T X 1 = K 2 T ( K − 1 1 K 1 ) X 1 = w 1 K 2 T K − 1 w 2 y 2 y 1 4 4 5 1 | {z } 1 1 3 × 3 matrix And finally: 2 3 2 3 2 3 x 2 a b c x 1 5 = w 2 y 2 d e f y 1 4 4 5 4 5 1 1 g h i Sanja Fidler CSC420: Intro to Image Understanding 6 / 1

Homography The nice thing about homography is that once we have it, we can compute where any point from one projective plane maps to on the second projective plane. We do not need to know the 3D location of that point. We don’t even need to know the camera parameters. We still owe one more explanation for Lecture 9. Sanja Fidler CSC420: Intro to Image Understanding 7 / 1

Homography The nice thing about homography is that once we have it, we can compute where any point from one projective plane maps to on the second projective plane. We do not need to know the 3D location of that point. We don’t even need to know the camera parameters. We still owe one more explanation for Lecture 9. Sanja Fidler CSC420: Intro to Image Understanding 7 / 1

Remember Panorama Stitching from Lecture 9? [Source: Fernando Flores-Mangas] Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

Remember Panorama Stitching from Lecture 9? Each pair of images is related by homography. Why? [Source: Fernando Flores-Mangas] Sanja Fidler CSC420: Intro to Image Understanding 8 / 1

Rotating the Camera Rotating my camera with R is the same as rotating the 3D points with R T (inverse of R ): X 2 = R T X 1 where X 1 is a 3D point in the coordinate system of the first camera and X 2 the 3D point in the coordinate system of the rotated camera. We can use the same trick as before, where we have T = R : 2 3 2 3 x 1 x 2 5 = K X 1 5 = K X 2 and w 1 y 1 w 2 y 2 4 4 1 1 2 3 2 3 x 2 x 1 5 = w 1 K R K − 1 y 2 y 1 w 2 4 4 5 | {z } 1 1 3 × 3 matrix Sanja Fidler CSC420: Intro to Image Understanding 9 / 1