and Retrieval Source: H. Jegou Source: H. Jegou Source: H. Jegou - PowerPoint PPT Presentation

Harris detector Hessian determinant • Second moment matrix / autocorrelation matrix     2 I I I ( ) ( )       x D x y D   g ( , ) ( )   I D I 2   I I I ( ) ( )   x y D y D I y I y I x 1. Image derivatives g x (  D ), g y (  D ), I x I y I x 2 I y 2 I x 2 2. Square of derivatives

Harris detector Hessian determinant • Second moment matrix / autocorrelation matrix     2 I I I ( ) ( )       x D x y D   g ( , ) ( ) I x I y   I D I 2   I I I ( ) ( )   1. Image x y D y D derivatives I y 2 I x I y I x 2 2. Square of derivatives 3. Gaussian filter g(  I ) g(I x 2 ) g(I y 2 ) g(I x I y )

Harris detector Hessian determinant • Second moment matrix / autocorrelation matrix I x I y 1. Image derivatives I y 2 I x I y I x 2 2. Square of derivatives 3. Gaussian filter g(  I ) g(I x 2 ) g(I y 2 ) g(I x I y ) 4. Cornerness function – both eigenvalues are strong           har det[ ( , )] [trace( ( , ))] I D I D     2 2 2 2 2 2 g I g I g I I g I g I ( ) ( ) [ ( )] [ ( ) ( )] x y x y x y 5. Non-maxima suppression har

Harris detector How to select features? large small

Harris detector How to select features? large large

Harris detector How to select features? small small

Harris detector Interpretation of eigenvalues Classification of image  2 points using “Edge” eigenvalues of M:  2 >>  1 “Corner”  1 and  2 are large,  1 ~  2 ; E increases in all directions  1 and  2 are small; E is almost constant “Edge” “Flat” in all directions  1 >>  2 region  1 Source: K. Grauman

Harris detector Corner response function Measure of corner response: Does not require computing of the eigenvalues α - constant α = 0.04 - 0.06

Harris detector Corner response function  2 “Edge” R < 0 “Corner” R > 0 | R| small “Edge” “Flat” R < 0 region  1 Source: K. Grauman

Harris detector Algorithm • Compute M matrix within all image windows to get their R scores • Find points with large corner response (R > threshold) • Take the points of local maxima of R

Harris detector Algorithm Source: D. Frolova

Harris detector Algorithm Compute corner response R Source: D. Frolova

Harris detector Algorithm Find points with large corner response: R>threshold Source: D. Frolova

Harris detector Algorithm Take only the points of local maxima of R Source: D. Frolova

Harris detector Algorithm Source: D. Frolova

Harris detector Typical response Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial

Properties of ideal feature • Local: features are local, so robust to occlusion and clutter (no prior segmentation) • Invariant (or covariant) Remember this? • Robust: noise, blur, discretization, compression, etc. do not have a big impact on the feature • Distinctive: individual features can be matched to a large database of objects • Quantity: many features can be generated for even small objects • Accurate: precise localization • Efficient: close to real-time performance

Is the Harris corner detector rotation invariant? Ellipse rotates but its shape remains the same Corner response R is invariant to image rotation

Is the Harris corner detector scale invariant?

Is the Harris corner detector scale invariant? Not invariant to image scale! All points will be This is a corner classified as edges

How can we detect scale invariant interest points?

Exhaustive search A multi-scale approach Source: T. Tuytelaars

Exhaustive search A multi-scale approach Source: T. Tuytelaars

Exhaustive search A multi-scale approach Source: T. Tuytelaars

Exhaustive search A multi-scale approach Source: T. Tuytelaars

Exhaustive search A multi-scale approach Extract patch from each image individually Source: T. Tuytelaars

Scale invariant detection We want to extract the patches from each image independently. Source: T. Tuytelaars

Exhaustive search Solution Design a function on the region, which is “scale invariant” ( the same for corresponding regions, even if they are at different scales ) Example: average intensity. For corresponding regions (even of different sizes) it will be the same. - For a point in one image, we can consider it as a function of region size (patch width) Image 1 f f Image 2 scale = 1/2 region size region size Lindeberg et. al , 1996

Scale invariant detection Common approach Take a local maximum of this function Observation: region size, for which the maximum is achieved, should be invariant to image scale. This scale invariant region size is found in each image independently ! Image 1 f f Image 2 scale = 1/2 s 1 s 2 region size region size Lindeberg et. al , 1996

Scale invariant detection Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial      f I x f I x ( ( , )) ( ( , ))   i i i i 1 m 1 m Same operator responses if the patch contains the same image up to scale factor How to find corresponding patch sizes?

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial    f I x ( ( , )) f I x ( ( , )) i  i i  i 1 m 1 m

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial    f I x ( ( , )) f I x ( ( , )) i  i i  i 1 m 1 m

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial    f I x ( ( , )) f I x ( ( , )) i  i i  i 1 m 1 m

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial    f I x ( ( , )) f I x ( ( , )) i  i i  i 1 m 1 m

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial    f I x ( ( , )) f I x ( ( , )) i  i i  i 1 m 1 m

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial     f I x ( ( , )) f I x ( ( , )) i  i i  i 1 m 1 m

Scale invariant detection Function responses for increasing scale (scale signature) Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial

Scale invariant detection Common approach Image 1 f f Image 2 scale = 1/2 s 1 s 2 region size region size

Scale invariant detection Common approach • A good function for scale detection has one stable sharp peak • For usual images: a good function would be one which responds to contrast (sharp local intensity change)

Scale invariant detection Common approach We define the characteristic scale that produces peak of Laplacian response Source: L. Lazebnik

Scale invariant detection Harris-Laplace Source: K. Grauman

Scale invariant detection Harris-Laplace Source: K. Grauman

Scale invariant detection Harris-Laplace Source: K. Grauman

Scale invariant detection Harris-Laplace Source: K. Grauman

Scale invariant detection Harris-Laplace Source: K. Grauman

Scale invariant detection Harris-Laplace Source: K. Grauman

Scale invariant detection Harris-Laplace Interest points (blobs) are local maxima in both position and scale scale  List of (x, y, σ ) Source: K. Grauman

Scale invariant detection Harris-Laplace Scale-space blob detector example Source: T. Lindeberg

Scale invariant detection Harris-Laplace Scale-space blob detector example Source: L. Lazebnik

Scale invariant detection Harris-Laplace Harris points vs. Harris Laplace points Harris points Harris-Laplace points Source: C. Schmid

Scale invariant detection Technical detail • Functions for determining scale f = Kernel * Image Kernels:        2 L G x y G x y ( , , ) ( , , ) xx yy (Laplacian)     DoG G x y k G x y ( , , ) ( , , ) (Difference of Gaussians) where Gaussian Lowe, 1999