? ( interest - PowerPoint PPT Presentation

ןיינע תודוקנ לט לש םיפקשה לע ססובמרנסה ?

םויהל םיאשונה • הנומתב תוניינעמ תודוקנ רותיא( interest point ) detection הנומתב תודוקנ רואת( • interest point ) description תודוקנ ירואת ןיב תומאתה תאיצמ( • interest point ) matching

תורוקמ • קרפ4 דומילה רפסב • סרוק לש דומילה רפס"הנומת דוביע " Gonzalez & Woods, “Digital Image Processing, 3 rd edition”

ןיינעמ עדימ אצמנ הנומתב הפיא?

Image matching by Diva Sian by swashford

Harder case by Diva Sian by scgbt

Harder still? NASA Mars Rover images

Answer below (look for tiny colored squares…) NASA Mars Rover images with SIFT feature matches Figure by Noah Snavely

Local features and alignment • We need to match (align) images • Global methods sensitive to occlusion, lighting, parallax effects. So look for local features that match well. • How would you do it by eye? [Darya Frolova and Denis Simakov]

Local features and alignment • Detect feature points in both images [Darya Frolova and Denis Simakov]

Local features and alignment • Detect feature points in both images • Find corresponding pairs [Darya Frolova and Denis Simakov]

Local features and alignment • Detect feature points in both images • Find corresponding pairs • Use these pairs to align images [Darya Frolova and Denis Simakov]

Local features and alignment • Problem 1: – Detect the same point independently in both images no chance to match! We need a repeatable detector [Darya Frolova and Denis Simakov]

Local features and alignment • Problem 2: – For each point correctly recognize the corresponding one ? We need an invariant and distinctive descriptor [Darya Frolova and Denis Simakov]

Geometric transformations

Photometric transformations Figure from T. Tuytelaars ECCV 2006 tutorial

And other nuisances… • Noise • Blur • Compression artifacts • …

Invariant local features Subset of local feature types designed to be invariant to common geometric and photometric transformations. Basic steps: 1) Detect distinctive interest points 2) Extract invariant descriptors 3) Match them! Figure: David Lowe

Main questions • Where will the interest points come from? – What are salient features that we’ll detect in multiple views? • How to describe a local region? • How to establish correspondences , i.e., compute matches?

Finding Corners • Corners are repeatable and distinctive • Key property: in the region around a corner, image gradient has two or more dominant directions C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference : 1988, pages 147--151. Source: Lana Lazebnik

סירה לש תוניפה רתאמ: הניפ יהמ? •תישאר , תרוכזת – טנאידרגה     f f      f I I       x y   x y Based on slides by Robert Collins, Pen State, 04

סירה לש תוניפה רתאמ: הניפ יהמ? •תישאר , תרוכזת – טנאידרגה     f f      f I I       x y   x y Based on slides by Robert Collins, Pen State, 04

סירה לש תוניפה רתאמ: הניפ יהמ? •תישאר , תרוכזת – טנאידרגה     f f      f I I       x y   x y Based on slides by Robert Collins, Pen State, 04

רוזיפ ןויפאםיטנאידרגה • הספילא תודוקנל םיאתנ( תוגלפתהתינאיסואג )תישארב הזכרמש •איה הניפ" :לודג לוגיעל המודה הספילא " בחרמבםיטנאידרגה •איה הניפ : תוגלפתהתינאיסואג בחרמב םיריצ ינשב ההובג תונוש םע םיטנאידרגה Based on slides by Robert Collins, Pen State, 04

תורבתסה תצק תצירטמ / ) • תרוזפת תצירטמ( scatter matrix ) תפתושמ תונוש( covariance matrix •יהת:   I I   x xn  1 2  A n w I I   y yn 12  n •ידי לע תרדגומ תפתושמה תונושה תצירטמ:     2 I I I    x x y T  M AA   2   I I I   x y y  2 2 הספילאה תא הרידגמ : םה םיימצעה היכרע • M הספילאה יריצ יכרא! !תישארה איה תיתימאה תוגלפתהה עצוממ יכ החנהה תחת

הרבגלא תצק :םיימצע םירוטקוו םיכרע x 1 x 2   1 2

הרבגלא תצק :םיימצע םירוטקוו םיכרע • The eigenvectors of a matrix A are the vectors x that satisfy: The scalar  is the eigenvalue corresponding to x • • The eigenvalues are found by solving: – In our case, A = M is a 2x2 matrix, so we have      M M 11 12   det 0    M M  21 22 – The solution:   1            2 M M M M M M 4     11 22 12 21 11 22 2 Once you know  , you find x by solving •        M M x 11 12     0    M M    y 21 22 Based on slides by Neel Joshi, UW, 10

ונניינעל הרזח •הניפ ונל שי יכ עדנ ךיא זא ? היכרע ינשב ןנובתנ לש םיימצעה M  2 “Edge”  2 >>  1 “Corner”  1 and  2 are large,  1 ~  2 “Edge” “Flat”  1 and  2 are small;  1 >>  2 region  1

סיראה לש תוניפה ההזמ •וא ,בושיחל ליעי רתוי:            2 2 R M M det( ) trace ( ) ( ) 1 2 1 2 α : constant (0.04 to 0.06) “Edge” R < 0 “Corner” R > 0 |R| small “Edge” “Flat” R < 0 region

Harris Corner Detector • Algorithm steps: – Compute M matrix within all image windows to get their R scores – Find points with large corner response ( R > threshold) – Take the points of local maxima of R

Harris Detector: Workflow

Harris Detector: Workflow Compute corner response R

Harris Detector: Workflow Find points with large corner response: R> threshold

Harris Detector: Workflow Take only the points of local maxima of R

Harris Detector: Workflow

Harris Detector: Properties • Rotation invariance Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation

Harris Detector: Properties • Not invariant to image scale All points will be Corner ! classified as edges

• How can we detect scale invariant interest points?

Option1: Exhaustive search • Multi-scale approach Slide from T. Tuytelaars ECCV 2006 tutorial

Option1: Exhaustive search • Multi-scale approach

Option1: Exhaustive search • Multi-scale approach Too Time consuming!

Scale Invariant Detection • Consider regions (e.g. circles) of different sizes around a point • Regions of corresponding sizes will look the same in both images

Scale Invariant Detection • The problem: how do we choose corresponding circles independently in each image?

Option 2: Invariance • Extract patch from each image individually But how?

Automatic scale selection • Solution: – Design a function on the region, which is “scale invariant” ( the same for corresponding regions, even if they are at different scales ) Example: average intensity. For corresponding regions (even of different sizes) it will be the same. – For a point in one image, we can consider it as a function of region size (patch width) Image 1 f f Image 2 scale = 1/2 region size region size

Automatic scale selection • Common approach: Take a local maximum of this function Observation: region size, for which the maximum is achieved, should be invariant to image scale. Important: this scale invariant region size is found in each image independently! Image 1 f f Image 2 scale = 1/2 s 1 s 2 region size region size

Automatic Scale Selection • Function responses for increasing scale (scale signature)    f I x ( ( , )) f I x ( ( , ))  i  i i i m m 1 1 K. Grauman, B. Leibe 48 K. Grauman, B. Leibe

Automatic Scale Selection • Function responses for increasing scale (scale signature)    f I x ( ( , )) f I x ( ( , ))  i  i i i m m 1 1 K. Grauman, B. Leibe 49 K. Grauman, B. Leibe

Automatic Scale Selection • Function responses for increasing scale (scale signature)    f I x ( ( , )) f I x ( ( , ))  i  i i i m m 1 1 K. Grauman, B. Leibe 50 K. Grauman, B. Leibe