Interest Point Detectors & RANSAC Instructor - Simon Lucey - PowerPoint PPT Presentation

Interest Point Detectors & RANSAC Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps

Today • Image Gradients. • Interest Point Detectors • Matching Regions. • RANSAC.

I I ( x, y ) I ( x + 1 , y + 1) Simoncelli & Olshausen 2001

I I ( x, y ) I ( x + 8 , y + 8) Simoncelli & Olshausen 2001

I I ( x, y ) I ( x + 16 , y + 16) Simoncelli & Olshausen 2001

I I ( x, y ) I ( x + 50 , y + 50) Simoncelli & Olshausen 2001

I I ( x, y ) I ( x + 50 , y + 50) Simoncelli & Olshausen 2001

Pixel Coherence • The pixel coherence assumption demonstrates that pixels within natural images are heavily correlated with one another within a local neighborhood (e.g., +/- 5 pixels). N • For example, 3 4 3 2 ? 5 3 5 4 2 5

Pixel Coherence • The pixel coherence assumption demonstrates that pixels within natural images are heavily correlated with one another within a local neighborhood (e.g., +/- 5 pixels). N • For example, “Typically assume that pixels within +/- 3, 5 or 7 pixels are highly correlated.” 3 4 3 2 ? 5 3 5 4 2 5

Estimating Gradients • Images are a discretely sampled representation of a continuous signal, I G x H x (Black) 6

Estimating Gradients • Images are a discretely sampled representation of a continuous signal, I G x H I G x H x x (Black) 7

Estimating Gradients • What if I want to know given that I have only the ∆ x appearance at and ? I ( x 0 ) I ( x 0 + ∆ x ) I G x H I F x H I ( x 0 + ∆ x ) I ( x 0 ) x 0 x x (Black) 8

Estimating Gradients • Again, simply take the Taylor series approximation? T I ( x 0 + ∆ x ) ≈ I ( x 0 ) + ∂ I ( x 0 ) ∆ x ∂ x I G x H I F x H I ( x 0 + ∆ x ) I ( x 0 ) x 0 x x (Black) 9

Estimating Gradients • Again, simply take the Taylor series approximation? T I ( x 0 + ∆ x ) ≈ I ( x 0 ) + ∂ I ( x 0 ) ∆ x ∂ x I G x H I F x H I ( x 0 + ∆ x ) I ( x 0 ) ∂ I ( x 0 ) ∂ x x 0 x x (Black) 9

Gradients through Filters • Traditional method for calculating gradients in vision is through the use of edge filters. (e.g., Sobel, Prewitt). ∗ � I x “Horizontal” ∗ � I y I “Vertical” ∂ I ( x ) • where, = [ � I x ( x ) , � I y ( x )] T ∂ x 10

Gradients through Filters • Traditional method for calculating gradients in vision is through the use of edge filters. (e.g., Sobel, Prewitt). ∗ � I x “Horizontal” ∗ � I y I “Vertical” “Often have to apply a smoothing filter as well.” ∂ I ( x ) • where, = [ � I x ( x ) , � I y ( x )] T ∂ x 10

Today • Natural Image Statistics. • Interest Point Detectors. • Matching Regions. • RANSAC.

Interest Point Detectors

Interest Point Detectors • Interest point detectors, in general, detect corners. • A dense approach (using all pixels) will be far too slow. • Matching lines suffers from the “aperture problem” (i.e. has the line moved along itself?). • Corners are:- • relatively sparse • reasonably cheap to compute • well-localized • appear quite robustly • good for matching • useful for geometry.

Interest Point Detectors • Approach is based on concept of auto-correlation, X || I ( x k , y k ) − I ( x k + ∆ x, y k + ∆ y ) || 2 A ( ∆ x, ∆ y ) = ( x k ,y k ) ∈ N ( x,y ) ( ∆ x, ∆ y ) N 14

Interest Point Detectors • Approach is based on concept of auto-correlation, X || I ( x k ) − I ( x k + ∆ x ) || 2 A ( ∆ x ) = x k ∈ N ( x ) “in vector form” N ( x ) “local neighborhood” 15

Interest Point Detectors • Which patches match the auto-correlation responses? X || I ( x k ) − I ( x k + ∆ x ) || 2 A ( ∆ x ) = x k ∈ N ( x ) 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 A ( ∆ x ) 16

Interest Point Detectors • Which patches match the auto-correlation responses? X || I ( x k ) − I ( x k + ∆ x ) || 2 A ( ∆ x ) = x k ∈ N ( x ) 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 A ( ∆ x ) 16

Harris Detector • Harris & Stephens (1988) proposed, 17

Harris Detector • How can one characterize these three situations? X || I ( x k ) − I ( x k + ∆ x ) || 2 A ( ∆ x ) = x k ∈ N ( x ) • We can simplify this through a linear approximation, T || I ( x i ) − I ( x i ) − ∂ I ( x i ) X ∆ x || 2 ≈ ∂ x i ∈ N ∆ x T H ∆ x ≈ • where, T ∂ I ( x i ) ∂ I ( x i ) X H = ∂ x ∂ x i ∈ N

Harris Corner Detector T ∂ I ( x i ) ∂ I ( x i ) X Make decision H = ∂ x ∂ x based on image i ∈ N structure tensor

Harris Detector 20

Effects of Scale • Consider a single row or column of the image – Plotting intensity as a function of position gives a signal Source: S. Seitz

Effects of Scale • Consider a single row or column of the image – Plotting intensity as a function of position gives a signal • Where is the edge? Source: S. Seitz

Search Multiple Scales f Source: S. Seitz

Search Multiple Scales f g Source: S. Seitz

Search Multiple Scales f g f * g Source: S. Seitz

Search Multiple Scales f g f * g Source: S. Seitz

Search Multiple Scales f g f * g • To find edges, look for peaks in Source: S. Seitz

SIFT Detector • Filter with difference of Gaussian filters at increasing scales • Build image stack (scale space) • Find extrema in this 3D volume

Coarse to Fine 24

SIFT Detector Remove those Identified Corners Remove those where contrast on edges is low

Today • Natural Image Statistics. • Interest Point Detectors. • Matching Regions. • RANSAC.

Matching Regions • Once we have detected local regions, we must then match them. ( x 2 , y 2 ) ( x 1 , y 1 ) image 2 image 1 SSD : sum of square difference X || I 1 ( x 1 + ∆ x, y 1 + ∆ y ) − I 2 ( x 2 + ∆ x, y 2 + ∆ y ) || 2 2 [ ∆ x, ∆ y ] T ∈ N Small difference values � similar patches 27

Problems with SSD • SSD measures are sensitive to both, • geometric and, I 1 • photometric variation. • Common practice to use descriptors. I 2 28

Planar Affine Patch Assumption • A number of technique have been proposed to detect affine covariant regions (Schmid et al. 2004). “View 1” “View 2” 29

Rotation Invariance • Estimation of the dominant orientation – extract gradient orientation – histogram over gradient orientation – peak in this histogram • Rotate patch in dominant direction

SIFT Descriptor 1. Compute image gradients 2. Pool into local histograms 3. Concatenate histograms 4. Normalize histograms More on this in future lectures.

Why Pool?

Why Pool?

Why Pool? “average”

Why Pool? “histogram” ∗ “pooling” “blurring”

Why Pool? “histogram” ∗ “pooling” “blurring”

MATLAB Example • Example in MATLAB, >> h = fspecial(‘gaussian’,[25,25],3); >> resp = imfilter(im, h); ∗ h resp

MATLAB Example • Example in MATLAB, >> h = fspecial(‘gaussian’,[25,25],3); >> resp = imfilter(im, h); ∗ h resp

Other Descriptors • Since SIFT, a plethora of other descriptors have been proposed. • SIFT is sometimes problematic to use in practice as it is • protected under existing patents. • In OpenCV alone there are, • ORB • SURF - (patent protected) • BRISK • BRIEF • FAST • PHREAK • See following link for tutorial in OpenCV.

Other Descriptors • Since SIFT, a plethora of other descriptors have been proposed. • SIFT is sometimes problematic to use in practice as it is • protected under existing patents. • In OpenCV alone there are, • ORB • SURF - (patent protected) • BRISK • BRIEF • FAST • PHREAK • See following link for tutorial in OpenCV.

Today • Natural Image Statistics. • Interest Point Detectors. • Matching Regions. • RANSAC.

Robust Estimation • Least squares criterion is not robust to outliers • For example, the two outliers here cause the fitted line to be quite wrong. • One approach to fitting under these circumstances is to use RANSAC – “Random sampling by consensus”

RANSAC

RANSAC

Fitting a homography with RANSAC Original images Initial matches Inliers from RANSAC

Things to try in iOS - GPUImage See - https://github.com/BradLarson/GPUImage for more details.

More to read… • Prince et al. • Chapter 13, Sections 2-3. • Corke et al. • Chapter 13, Section 3. • E. Rublee et al. “ORB: an efficient alternative to SIFT or SURF”, ICCV 2011.