I Instance-level recognition t l l iti Cordelia Schmid INRIA - PowerPoint PPT Presentation

I Instance-level recognition t l l iti Cordelia Schmid INRIA

Instance-level recognition Instance level recognition Particular objects and scenes large databases Particular objects and scenes, large databases …

Application pp Search photos on the web for particular places p p p Find these landmarks ...in these images and 1M more

Applications Applications • Take a picture of a product or advertisement Take a picture of a product or advertisement  find relevant information on the web [Google Goggles, Milpix Pixee]

Applications Applications • Copy detection for images and videos Search in 200h of video Query video

Difficulties Find the object despite – large changes in scale, viewpoint, lighting – crop and occlusion – not much texture/structure not much texture/structure  requires local invariant descriptors Scale Viewpoint Occlusion Lighting

Difficulties Difficulties • Very large images collection  need for efficient indexing Very large images collection  need for efficient indexing – Flickr has 2 billions photographs, more than 1 million added daily Flickr has 2 billions photographs more than 1 million added daily – Facebook has 15 billions images (~27 million added daily) – Facebook has 15 billions images ( 27 million added daily) – Large personal collections Large personal collections – Video collections with a large number of videos, i.e., YouTube g

Approach: matching local invariant descriptors Approach: matching local invariant descriptors I Image content is transformed into local features that are i f d i l l f h invariant to geometric and photometric transformations Local Features Local Features, e.g. SIFT [Lowe04] 8 K. Grauman, B. Leibe S lide credit: David Lowe

Overview Overview • Local invariant features • Matching and recognition with local features • Efficient visual search • Very large scale search

Local features Local features ( ) ( ) local descriptor Several / many local descriptors per image Robust to occlusion/clutter + no object segmentation required Robust to occlusion/clutter + no object segmentation required Photometric : distinctive Invariant : to image transformations + illumination changes

Local features: interest points Local features: interest points

Local features: Contours/segments Local features: Contours/segments

Local features: segmentation Local features: segmentation

Local features Local features 1) Extraction of local features 1) Extraction of local features – Contours/segments – Interest points & regions Interest points & regions – Regions by segmentation – Dense features, points on a regular grid 2) Description of local features – Dependant on the feature type – Contours/segments  angles, length ratios – Interest points  greylevels, gradient histograms I t t i t  l l di t hi t – Regions (segmentation)  texture + color distributions

Line matching Line matching • Extraction de contours Extraction de contours – Zero crossing of Laplacian – Local maxima of gradients Local maxima of gradients • Chain contour points (hysteresis) p ( y ) • Extraction of line segments g • Description of segments Description of segments – Mi-point, length, orientation, angle between pairs etc.

Experimental results – line segments Experimental results line segments images 600 x 600

Experimental results – line segments Experimental results line segments 248 / 212 line segments extracted

Experimental results – line segments Experimental results line segments 89 matched line segments - 100% correct

Experimental results – line segments Experimental results line segments 3D reconstruction

Problems of line segments Problems of line segments • Often only partial extraction Often only partial extraction – Line segments broken into parts – Missing parts Missing parts • Information not very discriminative Information not very discriminative – 1D information – Similar for many segments • Potential solutions – Pairs and triplets of segments – Interest points

Example results - interest points Example results interest points I t Interest points extracted with Harris detector (~ 500 points) t i t t t d ith H i d t t ( 500 i t )

Matching interest points Matching interest points Find corresponding locations in the image Find corresponding locations in the image

Matching interest points Matching interest points Matching Matching I t Interest points matched based on cross-correlation (188 pairs) t i t t h d b d l ti (188 i )

Matching interest points Matching interest points constrai constrai Global constraint Global constraint - Robust estimation of the fundamental matrix Robust estimation of the fundamental matrix 99 inliers 99 inliers 89 outliers 89 outliers

Application: Panorama stitching pp g Images courtesy of A. Zisserman.

Overview Overview • Harris interest points + SSD, ZNCC, SIFT • Scale & affine invariant interest point detectors • Evaluation and comparison of different detectors • Region descriptors and their performance

Harris detector [Harris & Stephens’88] Harris detector [Harris & Stephens 88] B Based on the idea of auto-correlation d th id f t l ti I Important difference in all directions => interest point t t diff i ll di ti i t t i t

Harris detector Harris detector  x  y Auto-correlation function for a point and a shift x y ( , ) ( , )           A A x y I I x y I I x x y y 2 2 ( ( , ) ) ( ( ( ( , ) ) ( ( , )) )) k k k k  x y W x y ( , ) ( , ) k k   x  x  y y ( ( , ) ) W

Harris detector Harris detector  x  y Auto-correlation function for a point and a shift x y ( , ) ( , )           A A x y I I x y I I x x y y 2 2 ( ( , ) ) ( ( ( ( , ) ) ( ( , )) )) k k k k  x y W x y ( , ) ( , ) k k   x  x  y y ( ( , ) ) W → uniform region small in all directions A { y { x y → contour large in one directions g ( ( , , ) ) → interest point large in all directions

Harris detector Harris detector

Harris detector Harris detector Discret shifts are avoided based on the auto-correlation matrix with first order approximation   x                I I x x x x y y y y I I x x y y I I x x y y I I x x y y ( ( , ) ) ( ( , ) ) ( ( ( ( , ) ) ( ( , )) ))      k k k k x k k y k k y         A x y I x y I x x y y 2 ( , ) ( ( , ) ( , )) k k k k  x y W x y ( , ) ( , ) k k  2     x         I x y I x y ( , ) ( , )       x k k y k k y y          x x y y W W ( ( , ) ) k k k k

Harris detector Harris detector     I x y I x y I x y 2 ( ( , )) ( , ) ( , )   x k k x k k y k k    x x                  x y W W x y W W x y ( ( , ) ) ( ( , ) ) k k k k        I x y I x y I x y 2 y ( , ) ( , ) ( ( , ))     x k k y k k y k k     x y W x y W ( , ) ( , ) k k k k Auto-correlation matrix the sum can be smoothed with a Gaussian the sum can be smoothed with a Gaussian       I 2 I I x                x y G G x x x x y y        I I I 2 y       x y y

Harris detector Harris detector • Auto-correlation matrix Auto correlation matrix   I 2 I I   x x x x y y A x y G     ( ( , ) ) I I I 2     x y y – captures the structure of the local neighborhood – measure based on eigenvalues of this matrix measure based on eigenvalues of this matrix => interest point • 2 strong eigenvalues => contour • 1 strong eigenvalue • 0 eigenvalue => uniform region

Interpreting the eigenvalues Interpreting the eigenvalues Classification of image points using eigenvalues of autocorrelation matrix :  2 “Edge”  2 >>  1  2 >>  1 “Corner”  1 and  2 are large,  1 ~  2 ;   \  1 and  2 are small; “Edge” “Flat”  1 >>  2 region  1

Corner response function Corner response function            R A A 2 2 det( ( ) ) trace ( ( ) ) ( ( ) ) 1 1 2 2 1 1 2 2 α : constant (0.04 to 0.06) “Edge” R < 0 R < 0 “Corner” “Corner” R > 0 |R| small “Edge” “Flat” R < 0 region