Today Alignment & warping 2d transformations Forward and - PDF document

Today • Alignment & warping – 2d transformations – Forward and inverse image warping – Constructing mosaics • Homographies Alignment & Warping • Robust fitting with RANSAC Thursday, Oct 9 Kristen Grauman UT-Austin Main questions Motivation: Recognition Warping: Given a T source image and a transformation, what does the transformed output look like? Alignment: Given two T images with corresponding features, what is the transformation between them? Figures from David Lowe Motivation: medical image Motivation: Mosaics registration • Getting the whole picture – Consumer camera: 50 ˚ x 35 ˚ Slide from Brown & Lowe 2003

Motivation: Mosaics Motivation: Mosaics • Getting the whole picture • Getting the whole picture – Consumer camera: 50 ˚ x 35 ˚ – Consumer camera: 50 ˚ x 35 ˚ – Human Vision: 176 ˚ x 135 ˚ – Human Vision: 176 ˚ x 135 ˚ • Panoramic Mosaic = up to 360 x 180° Slide from Brown & Lowe 2003 Slide from Brown & Lowe 2003 Parametric (global) warping Warping problem Examples of parametric warps: • Given a set of points and a transformation, generate the warped image aspect translation rotation T ( x,y ) y y’ x x’ f ( x,y ) g ( x’,y’ ) perspective affine Figure Alyosha Efros Source: Alyosha Efros Parametric (global) warping Scaling Scaling a coordinate means multiplying each of its components by a scalar T Uniform scaling means this scalar is the same for all components: p = (x,y) p’ = (x’,y’) Transformation T is a coordinate-changing machine: p’ = T (p) What does it mean that T is global ? • Is the same for any point p • can be described by just a few numbers (parameters) × 2 Let’s represent T as a matrix: p’ = M p ⎡ ⎤ ⎡ ⎤ x ' x = ⎢ ⎥ M ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ y ' y Source: Alyosha Efros Source: Alyosha Efros

Scaling Scaling Non-uniform scaling : different scalars per component: = x ' ax Scaling operation: = ' y by Or, in matrix form: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x ' a 0 x = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ X × 2, ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ y ' 0 b y Y × 0.5 scaling matrix S Source: Alyosha Efros Source: Alyosha Efros What transformations can be What transformations can be represented with a 2x2 matrix? represented with a 2x2 matrix? 2D Scaling? 2D Mirror about Y axis? = ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ s 0 x ' s * x x ' x = x x ⎢ ⎥ ⎡ ⎤ ⎡− ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ = − x ' 1 0 x x ' x = = ⎣ ⎦ 0 s ⎣ ⎦ y ' ⎣ ⎦ y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ y ' s * y = ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ y y ' 0 1 y y ' y y 2D Rotate around (0,0)? 2D Mirror over (0,0)? = Θ − Θ ⎡ ⎤ ⎡ Θ − Θ ⎤ ⎡ ⎤ x ' cos sin x x ' cos * x sin * y = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − = Θ + Θ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ = − x ' 1 0 x y ' sin * x cos * y Θ Θ x ' x = ⎣ y ' ⎦ ⎣ sin cos ⎦ ⎣ y ⎦ = − ⎢ ⎥ ⎢ − ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ y ' y y ' 0 1 y 2D Shear? 2D Translation? ⎡ ⎤ = + ⎡ ⎤ ⎡ ⎤ 1 sh x ' x sh * y x ' x x = x ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = + x ' x t = + y sh x y ⎣ ⎦ sh 1 ⎣ ⎦ ' * y ' ⎣ ⎦ y x NO! y y = + y ' y t y Source: Alyosha Efros Source: Alyosha Efros 2D Linear Transformations Homogeneous Coordinates Q: How can we represent translation as a 3x3 matrix ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x ' a b x = using homogeneous coordinates? ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ y ' c d y = + x ' x t x = + y ' y t Only linear 2D transformations can be represented with a 2x2 y matrix. Linear transformations are combinations of … • Scale, • Rotation, • Shear, and • Mirror Source: Alyosha Efros Source: Alyosha Efros

Homogeneous Coordinates Translation Q: How can we represent translation as a 3x3 matrix Homogeneous Coordinates using homogeneous coordinates? = + x ' x t ⎡ ⎤ + x ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x ' 1 0 t x x t x x = + ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ y y t ' = = + y ' 0 1 t y y t y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ y y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ 1 ⎣ 0 0 1 ⎦ 1 1 A: Using the rightmost column: ⎡ ⎤ t 1 0 x ⎢ ⎥ = T ranslation 0 1 t ⎢ ⎥ y ⎢ ⎥ 0 0 1 ⎣ ⎦ t x = 2 t y = 1 Source: Alyosha Efros Source: Alyosha Efros Basic 2D Transformations 2D Affine Transformations Basic 2D transformations as 3x3 matrices ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x ' a b c x ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ y ' d e f y x ' 1 0 t x x ' s 0 0 x x x ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = = ⎣ w ⎦ ⎣ 0 0 1 ⎦ ⎣ w ⎦ y ' 0 s 0 y y ' 0 1 t y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ y y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 1 ⎦ ⎣ 0 0 1 ⎦ ⎣ 1 ⎦ ⎣ 1 ⎦ ⎣ 0 0 1 ⎦ ⎣ 1 ⎦ Translate Scale Affine transformations are combinations of … • Linear transformations, and Θ − Θ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x ' cos sin 0 x x ' 1 sh 0 x • Translations x ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = Θ Θ = y ' sin cos 0 y y ' sh 1 0 y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ Parallel lines remain parallel ⎣ 1 ⎦ ⎣ 0 0 1 ⎦ ⎣ 1 ⎦ ⎣ 1 ⎦ ⎣ 0 0 1 ⎦ ⎣ 1 ⎦ Rotate Shear Source: Alyosha Efros Projective Transformations Today ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x ' a b c x ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = • Alignment & warping y ' d e f y ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ w ' ⎦ ⎣ g h i ⎦ ⎣ w ⎦ – 2d transformations Projective transformations: – Forward and inverse image warping • Affine transformations, and – Constructing mosaics • Projective warps • Homographies Parallel lines do not necessarily remain parallel • Robust fitting with RANSAC

Image warping Forward warping T ( x,y ) T ( x,y ) y y’ y y’ x x’ x x’ f ( x,y ) g ( x’,y’ ) f ( x,y ) g ( x’,y’ ) Send each pixel f ( x,y ) to its corresponding location Given a coordinate transform and a source image ( x’,y’) = T ( x,y ) in the second image f ( x,y ), how do we compute a transformed Q: what if pixel lands “between” two pixels? image g(x’,y’) = f ( T ( x,y ))? Slide from Alyosha Efros, CMU Slide from Alyosha Efros, CMU Forward warping Inverse warping T ( x,y ) T -1 ( x,y ) y y’ y y’ x x’ x x x’ f ( x,y ) g ( x’,y’ ) f ( x,y ) g ( x’,y’ ) Send each pixel f ( x,y ) to its corresponding location Get each pixel g ( x’,y’ ) from its corresponding location ( x’,y’) = T ( x,y ) in the second image ( x,y ) = T -1 ( x’,y’ ) in the first image Q: what if pixel lands “between” two pixels? Q: what if pixel comes from “between” two pixels? A: distribute color among neighboring pixels (x’,y’) – Known as “splatting” Slide from Alyosha Efros, CMU Slide from Alyosha Efros, CMU Inverse warping Bilinear interpolation Sampling at f(x,y): T -1 ( x,y ) y y’ x x x’ f ( x,y ) g ( x’,y’ ) Get each pixel g ( x’,y’ ) from its corresponding location ( x,y ) = T -1 ( x’,y’ ) in the first image Q: what if pixel comes from “between” two pixels? A: Interpolate color value from neighbors – nearest neighbor, bilinear… >> help interp2 Slide from Alyosha Efros, CMU Slide from Alyosha Efros, CMU