Image alignment Slides from Derek Hoiem, Svetlana Lazebnik Image - PowerPoint PPT Presentation

Image alignment Slides from Derek Hoiem, Svetlana Lazebnik Image source

Alignment applications • A look into the past

Alignment applications • A look into the past

Alignment applications • Cool video

Alignment applications Instance recognition T. Weyand and B. Leibe, Visual landmark recognition from Internet photo collections: A large-scale evaluation, CVIU 2015

Alignment challenges Small degree of overlap Intensity changes Occlusion, clutter, viewpoint change

Image Alignment Slide from L. Lazebnik.

Robust feature-based alignment Slide from L. Lazebnik.

Robust feature-based alignment • Extract features Slide from L. Lazebnik.

Robust feature-based alignment • Extract features • Compute putative matches Slide from L. Lazebnik.

Robust feature-based alignment • Extract features • Compute putative matches • Loop: • Hypothesize transformation T Slide from L. Lazebnik.

Robust feature-based alignment • Extract features • Compute putative matches • Loop: • Hypothesize transformation T • Verify transformation (search for other matches consistent with T ) Slide from L. Lazebnik.

Generating putative correspondences ? Slide from L. Lazebnik.

Generating putative correspondences ? ? ( ) ( ) = feature feature descriptor descriptor • Need to compare feature descriptors of local patches surrounding interest points Slide from L. Lazebnik.

Feature descriptors • Recall: feature detection and description Slide from L. Lazebnik.

Feature descriptors • Simplest descriptor: vector of raw intensity values • How to compare two such vectors? • Sum of squared differences (SSD) ( ) å 2 = - SSD( u, v ) u v i i i – Not invariant to intensity change • Normalized correlation å - - ( u u )( v v ) - - ( u u ) ( v v ) i i r = × = ( u, v ) i - - || u u || || v v || æ ö æ ö å å ç ÷ ç ÷ - - 2 2 ( u u ) ( v v ) ç ÷ ç ÷ j j è ø è ø j j – Invariant to affine intensity change Slide from L. Lazebnik.

Disadvantage of intensity vectors as descriptors • Small deformations can affect the matching score a lot Slide from L. Lazebnik.

Feature descriptors: SIFT • Descriptor computation: • Divide patch into 4x4 sub-patches • Compute histogram of gradient orientations (8 reference angles) inside each sub-patch • Resulting descriptor: 4x4x8 = 128 dimensions David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004. Slide from L. Lazebnik.

Feature descriptors: SIFT • Descriptor computation: • Divide patch into 4x4 sub-patches • Compute histogram of gradient orientations (8 reference angles) inside each sub-patch • Resulting descriptor: 4x4x8 = 128 dimensions • Advantage over raw vectors of pixel values • Gradients less sensitive to illumination change • Pooling of gradients over the sub-patches achieves robustness to small shifts, but still preserves some spatial information David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004. Slide from L. Lazebnik.

Feature matching • Generating putative matches : for each patch in one image, find a short list of patches in the other image that could match it based solely on appearance ? Slide from L. Lazebnik.

Problem: Ambiguous putative matches Source: Y. Furukawa

Rejection of unreliable matches • How can we tell which putative matches are more reliable? • Heuristic: compare distance of nearest neighbor to that of second nearest neighbor • Ratio of closest distance to second-closest distance will be high for features that are not distinctive Threshold of 0.8 provides good separation David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004. Slide from L. Lazebnik.

RANSAC • The set of putative matches contains a very high percentage of outliers RANSAC loop: 1. Randomly select a seed group of matches 2. Compute transformation from seed group 3. Find inliers to this transformation 4. If the number of inliers is sufficiently large, re-compute least-squares estimate of transformation on all of the inliers Keep the transformation with the largest number of inliers Slide from L. Lazebnik.

Robust feature-based alignment • Extract features • Compute putative matches • Loop: • Hypothesize transformation T • Verify transformation (search for other matches consistent with T ) Slide from L. Lazebnik.

Homography Example Camera Center Slide from A. Efros, S. Seitz, D. Hoiem

Common transformations original Transformed aspect rotation translation perspective affine Slide from A. Efros, S. Seitz, D. Hoiem

Scaling Scaling a coordinate means multiplying each of its components by a scalar Uniform scaling means this scalar is the same for all components: ´ 2 Slide from A. Efros, S. Seitz, D. Hoiem

Scaling Non-uniform scaling : different scalars per component: X ´ 2, Y ´ 0.5 Slide from A. Efros, S. Seitz, D. Hoiem

Scaling = x ' ax Scaling operation: = y ' by Or, in matrix form: é ù é ù é ù x ' a 0 x = ê ú ê ú ê ú y ' 0 b y ë û ë û ë û scaling matrix S Slide from A. Efros, S. Seitz, D. Hoiem

2-D Rotation (x’, y’) (x, y) x’ = x cos( q ) - y sin( q ) q y’ = x sin( q ) + y cos( q ) Slide from A. Efros, S. Seitz, D. Hoiem

2-D Rotation Polar coordinates… x = r cos ( f ) y = r sin ( f ) x’ = r cos ( f + q ) (x’, y’) y’ = r sin ( f + q ) Trig Identity… (x, y) x’ = r cos( f ) cos( q ) – r sin( f ) sin( q ) y’ = r sin( f ) cos( q ) + r cos( f ) sin( q ) q f Substitute… x’ = x cos( q ) - y sin( q ) y’ = x sin( q ) + y cos( q ) Slide from A. Efros, S. Seitz, D. Hoiem

2-D Rotation This is easy to capture in matrix form: ( ) ( ) q - q é ù é ù é ù x ' cos sin x = ê ú ê ú ê ú ( ) ( ) q q y ' sin cos y ë û ë û ë û R Even though sin( q ) and cos( q ) are nonlinear functions of q , • x’ is a linear combination of x and y • y’ is a linear combination of x and y What is the inverse transformation? Rotation by – q • - 1 = T R R • For rotation matrices Slide from A. Efros, S. Seitz, D. Hoiem

Basic 2D transformations é ù s 0 é x ' ù é x ù a é ù é ù 1 é ù x ' x x = ê ú x ê ú ê ú = ê ú ê ú ê ú 0 s y ' y a ë û ë û 1 ë û y ' y ë û ë û ë û y y Scale Shear é x ù ¢ é ù Q - Q 1 0 t é ù é ù é ù é x ù x ' cos sin x ê ú x = = y ê ú ê ú ê ú ê ú ê ú ê ú ¢ Q Q y ' sin cos y 0 1 t y ë û ë û ë û ë û ë û y ê ú 1 ë û Rotate Translate é x ù ¢ é ù é ù x a b c ê ú Affine is any combination of = y ê ú ê ú ê ú ¢ y d e f translation, scale, rotation, ë û ë û ê ú 1 ë û shear Affine Slide from A. Efros, S. Seitz, D. Hoiem

Affine Transformations é ù x ¢ é ù é ù x a b c ê ú = y Affine transformations are combinations of ê ú ê ú ê ú ¢ y d e f ë û ë û • Linear transformations, and ê ú 1 ë û • Translations or Properties of affine transformations: • Lines map to lines é x ' ù é a b c ù é x ù • Parallel lines remain parallel ê ú ê ú ê ú = • Ratios are preserved y ' d e f y ê ú ê ú ê ú • Closed under composition ê ú ê ú ê ú 1 0 0 1 1 ë û ë û ë û Slide from L. Lazebnik.

Projective Transformations é ù é ù é ù x ' a b c x Projective transformations are combos of ê ú ê ú ê ú = y ' d e f y • Affine transformations, and ê ú ê ú ê ú w ' g h i w ë û ë û ë û • Projective warps Properties of projective transformations: • Lines map to lines • Parallel lines do not necessarily remain parallel • Ratios are not preserved • Closed under composition • Models change of basis • Projective matrix is defined up to a scale (8 DOF) Slide from L. Lazebnik.

2D image transformations Slide from R. Szeliski

Fitting an affine transformation • Assume we know the correspondences, how do we get the transformation? ( x i y , ) i ¢ ¢ ( x i y , ) i ¢ = + x Mx t i i ¢ é ù é ù é ù é ù x m m x t i 1 2 i = + 1 Want to find M , t to minimize ê ú ê ú ê ú ê ú ¢ y m m y t ë û ë û ë û ë û n i 3 4 i 2 å ¢ - - 2 || x Mx t || i i = i 1 Slide from L. Lazebnik.