Instance-level recognition: Local invariant features Cordelia - PowerPoint PPT Presentation

Instance-level recognition: Local invariant features Cordelia Schmid INRIA, Grenoble

Overview Overview • Introduction to local features • Harris interest points + SSD, ZNCC, SIFT H i i t t i t SSD ZNCC SIFT • Scale & affine invariant interest point detectors

Local features Local features ( ) ( ) local descriptor Several / many local descriptors per image Robust to occlusion/clutter + no object segmentation required Robust to occlusion/clutter + no object segmentation required Photometric : distinctive Invariant : to image transformations + illumination changes

Local features Local features Interest Points Contours/lines Region segments

Local features Local features Interest Points Contours/lines Region segments Patch descriptors, i.e. SIFT Mi-points, angles Color/texture histogram

Interest points / invariant regions Interest points / invariant regions Harris detector Scale/affine inv. detector presented in this lecture t d i thi l t

Contours / lines Contours / lines • Extraction de contours Extraction de contours – Zero crossing of Laplacian – Local maxima of gradients Local maxima of gradients • Chain contour points (hysteresis) , Canny detector p ( y ) , y • Recent detectors – Global probability of boundary ( gPb ) detector [Malik et al., UC Berkeley] – Structured forests for fast edge detection (SED) [Dollar and S f f f (S ) Zitnick] – student presentation

Regions segments / superpixels Regions segments / superpixels Simple linear iterative clustering (SLIC) Simple linear iterative clustering (SLIC) Normalized cut [Shi & Malik], Mean Shift [Comaniciu & Meer], ….

Application: matching Application: matching Find corresponding locations in the image Find corresponding locations in the image

Illustration – Matching Illustration Matching I t Interest points extracted with Harris detector (~ 500 points) t i t t t d ith H i d t t ( 500 i t )

Illustration – Matching Illustration Matching Matching Matching I t Interest points matched based on cross-correlation (188 pairs) t i t t h d b d l ti (188 i )

Illustration – Matching Illustration Global constraints Global constraints Matching Global constraint Global constraint - Robust estimation of the fundamental matrix Robust estimation of the fundamental matrix 99 inliers 99 inliers 89 outliers 89 outliers

Application: Panorama stitching pp g Images courtesy of A. Zisserman.

Application: Instance-level recognition Application: Instance level recognition Search for particular objects and scenes in large databases Search for particular objects and scenes in large databases …

Difficulties Finding the object despite possibly large changes in scale, viewpoint, lighting and partial occlusion l i i t li hti d ti l l i  requires invariant description S Scale l Viewpoint Lighting Occlusion

Difficulties Difficulties • Very large images collection  need for efficient indexing V l i ll ti  d f ffi i t i d i – Flickr has 2 billion photographs, more than 1 million added daily Fli k h 2 billi h t h th 1 illi dd d d il – Facebook has 15 billion images (~27 million added daily) Facebook has 15 billion images ( 27 million added daily) – Large personal collections – Large personal collections – Video collections, i.e., YouTube Video collections, i.e., YouTube

Applications pp Search photos on the web for particular places p p p Find these landmarks ...in these images and 1M more

Applications Applications • Take a picture of a product or advertisement Take a picture of a product or advertisement  find relevant information on the web

Applications Applications • Copy detection for images and videos Search in 200h of video Query video

Overview Overview • Introduction to local features • Harris interest points + SSD, ZNCC, SIFT H i i t t i t SSD ZNCC SIFT • Scale & affine invariant interest point detectors

Harris detector [Harris & Stephens’88] Harris detector [Harris & Stephens 88] B Based on the idea of auto-correlation d th id f t l ti I Important difference in all directions => interest point t t diff i ll di ti i t t i t

Harris detector Harris detector  x  y Auto-correlation function for a point and a shift x y ( , ) ( , )           A A x y I I x y I I x x y y 2 2 ( ( , ) ) ( ( ( ( , ) ) ( ( , )) )) k k k k  x y W x y ( , ) ( , ) k k   x  x  y y ( ( , ) ) W

Harris detector Harris detector  x  y Auto-correlation function for a point and a shift x y ( , ) ( , )           A A x y I I x y I I x x y y 2 2 ( ( , ) ) ( ( ( ( , ) ) ( ( , )) )) k k k k  x y W x y ( , ) ( , ) k k   x  x  y y ( ( , ) ) W → uniform region small in all directions A { y { x y → contour ( ( , , ) ) large in one directions g → interest point large in all directions

Harris detector Harris detector Discret shifts are avoided based on the auto-correlation matrix with first order approximation   x                I I x x x x y y y y I I x x y y I I x x y y I I x x y y ( ( , ) ) ( ( , ) ) ( ( ( ( , ) ) ( ( , )) ))      k k k k x k k y k k y         A x y I x y I x x y y 2 ( , ) ( ( , ) ( , )) k k k k  x y W x y ( , ) ( , ) k k 2      x         I x y I x y ( , ) ( , )       x k k y k k y y          x x y y W W ( ( , ) ) k k k k

Harris detector Harris detector     I x y 2 I x y I x y ( ( , )) ( , ) ( , )   x k k x k k y k k    x x                  x y W W x y W W ( ( , ) ) ( ( , ) ) x y k k k k        I x y I x y I x y 2 y ( , ) ( , ) ( ( , ))     x k k y k k y k k     x y W x y W ( , ) ( , ) k k k k Auto-correlation matrix the sum can be smoothed with a Gaussian the sum can be smoothed with a Gaussian    2    I I I x                x y G G x x x x y y        I I I 2 y       x y y

Harris detector Harris detector • Auto-correlation matrix Auto correlation matrix   2 I I I   x x x x y y A x y G     ( ( , ) ) 2 I I I     x y y – captures the structure of the local neighborhood – measure based on eigenvalues of this matrix measure based on eigenvalues of this matrix => interest point • 2 strong eigenvalues => contour • 1 strong eigenvalue • 0 eigenvalue => uniform region

Interpreting the eigenvalues Interpreting the eigenvalues Classification of image points using eigenvalues of autocorrelation matrix :  2 “Edge”  2 >>  1  2 >>  1 “Corner”  1 and  2 are large,  1 ~  2 ;   \  1 and  2 are small; “Edge” “Flat”  1 >>  2 region  1

Corner response function Corner response function            R A A 2 2 det( ( ) ) trace ( ( ) ) ( ( ) ) 1 1 2 2 1 1 2 2 α : constant (0.04 to 0.06) “Edge” R < 0 R < 0 “Corner” “Corner” R > 0 |R| small “Edge” “Flat” R < 0 region

Harris detector Harris detector • Cornerness function C f ti          R A k trace A 2 k 2 det( ( ) ) ( ( ( ( )) )) ( ( ) ) 1 1 2 2 1 1 2 2 R d Reduces the effect of a strong contour th ff t f t t • • Interest point detection Interest point detection – Treshold (absolut, relatif, number of corners) – Local maxima         f thresh x y neighbourh ood f x y f x y , 8 ( , ) ( , )

Harris Detector: Steps Harris Detector: Steps

Harris Detector: Steps Harris Detector: Steps Compute corner response R

Harris Detector: Steps Harris Detector: Steps Find points with large corner response: R> threshold

Harris Detector: Steps Harris Detector: Steps Take only the points of local maxima of R

Harris Detector: Steps Harris Detector: Steps

Harris detector: Summary of steps Harris detector: Summary of steps 1. Compute Gaussian derivatives at each pixel 2. Compute second moment matrix A in a Gaussian window around each pixel i d d h i l 3. Compute corner response function R 4. Threshold R 5. Find local maxima of response function (non-maximum suppression) i )