Local invariant feature Would like discussion section, more review - PDF document

CS 376: Computer Vision - lecture 12 2/27/2018 Survey feedback • Generally like – Assignments – Topics – Lecture engaging, like examples, interactive nature • Lecture can be fast Local invariant feature – Would like discussion section, more review – Careful about tangential questions detection – Questions are on slides but answers not written there too – Would like to videotape lectures for review later • Content: Tues Feb 27 – Programming (would like more) vs. math (difficult) • Grading: make sure fair partial credit Kristen Grauman • Book can be difficult to follow • Website: UT Austin – Add direct link to current lecture (we have this) – Add TA emails (now added) • My office hours – Schedule with me if you can’t make standard window Segments as primitives for recognition Review: Segmentation with texture Multiple segmentations • Find “textons” by clustering vectors of filter bank outputs • Describe texture in a window based on texton histogram Count Image Texton map Texton index Count Count Texton index Texton index B. Russell et al., “Using Multiple Segmentations to Discover Objects and their Extent in Image Collections,” CVPR 2006 Malik, Belongie, Leung and Shi. IJCV 2001. Adapted from Lana Lazebnik Slide credit: Lana Lazebnik Category-agnostic object “proposals” Top-down segmentation Selective search for object recognition. Uijlings E. Borenstein and S. Ullman, “Class-specific, top-down segmentation,” ECCV 2002 A. Levin and Y. Weiss, “Learning to Combine Bottom-Up and Top-Down Segmentation,” et al. IJCV 2013. ECCV 2006. Slide credit: Kristen Grauman Slide credit: Lana Lazebnik 1

CS 376: Computer Vision - lecture 12 2/27/2018 Top-down segmentation Joint segmentation and recognition Normalized cuts Top-down segmentation Mask R-CNN, K. He et al., ICCV 2017 E. Borenstein and S. Ullman, “Class-specific, top-down segmentation,” ECCV 2002 A. Levin and Y. Weiss, “Learning to Combine Bottom-Up and Top-Down Segmentation,” ECCV 2006. Slide credit: Lana Lazebnik Video object segmentation Interactive image and video segmentation Goal : Extract all foreground objects Results achieved with average of 2 user clicks  even those unseen during training  without manual intervention. http://vision.cs.utexas.edu/projects/fusionseg/ S. Jain et al., FusionSeg: Learning to combine motion and appearance for [Jain & Grauman, HCOMP 2016] Click Carving fully automatic segmentation of generic objects in videos, CVPR 2017 https://github.com/suyogduttjain/click_carving Slide credit: Kristen Grauman Slide credit: Kristen Grauman Previously: Grouping & fitting Previously: Features and filters [fig from Shi et al] Clustering, segmentation, Transforming and fitting; what parts describing images; belong together? textures, colors, edges Slide credit: Kristen Grauman Slide credit: Kristen Grauman 2

CS 376: Computer Vision - lecture 12 2/27/2018 Important tool for multiple views: Local features Now: Multiple views Matching, invariant features, stereo vision, instance recognition Multi-view matching relies on local feature Lowe correspondences. Hartley and Zisserman Fei-Fei Li How to detect which local features to match? Slide credit: Kristen Grauman Local features: main components Local features: desired properties 1) Detection: Identify the interest points • Repeatability – The same feature can be found in several images despite geometric and photometric transformations 2) Description :Extract vector • Saliency  x [ x ( 1 ) , , x ( 1 ) ]  feature descriptor 1 1 d – Each feature has a distinctive description surrounding each interest • Compactness and efficiency point. – Many fewer features than image pixels • Locality  x [ x ( 2 ) , , x ( 2 ) ]  – A feature occupies a relatively small area of the 2 1 d 3) Matching: Determine image; robust to clutter and occlusion correspondence between descriptors in two views Slide credit: Kristen Grauman Goal: interest operator repeatability Goal: descriptor distinctiveness • We want to detect (at least some of) the • We want to be able to reliably determine same points in both images. which point goes with which. ? No chance to find true matches! • Must provide some invariance to geometric • Yet we have to be able to run the detection and photometric differences between the two procedure independently per image. views. 3

CS 376: Computer Vision - lecture 12 2/27/2018 Local features: main components 1) Detection: Identify the interest points 2) Description :Extract vector feature descriptor surrounding each interest point. 3) Matching: Determine correspondence between • What points would you choose? descriptors in two views Slide credit: Kristen Grauman Slide credit: Kristen Grauman Detecting corners Detecting corners Compute “cornerness” response at every pixel. Slide credit: Kristen Grauman Slide credit: Kristen Grauman Detecting local invariant Detecting corners features • Detection of interest points – Harris corner detection – Scale invariant blob detection: LoG • (Next time: description of local patches) Slide credit: Kristen Grauman 4

CS 376: Computer Vision - lecture 12 2/27/2018 Corners as distinctive interest points Corners as distinctive interest points We should easily recognize the point by  I I I I    looking through a small window x x x y M w ( x , y )   I I I I Shifting a window in any direction should give   x y y y a large change in intensity 2 x 2 matrix of image derivatives (averaged in neighborhood of a point). “flat” region: “edge”: “corner”:     I I I I    no change in no change significant I x I y I I Notation: x y     x y x y all directions along the edge change in all direction directions Slide credit: Alyosha Efros, Darya Frolova, Denis Simakov What does this matrix reveal? What does this matrix reveal? First, consider an axis-aligned corner: First, consider an axis-aligned corner:    2   I I I 0   x x y  1 M      2 I I I 0       x y y 2 This means dominant gradient directions align with x or y axis Look for locations where both λ’s are large. If either λ is close to 0, then this is not corner-like. What if we have a corner that is not aligned with the image axes? What does this matrix reveal? Corner response function    0  1 Since M is symmetric, we have M X X T    0   2   Mx x i i i “edge”: “corner”: “flat” region  1 >>  2  1 and  2 are large,  1 and  2 are  1 ~  2 ; small;  2 >>  1 The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal Cornerness score gradient directions in the window. (other variants possible) 5

CS 376: Computer Vision - lecture 12 2/27/2018 Harris corner detector Harris Detector: Steps 1) Compute M matrix for each image window to get their cornerness scores. 2) Find points whose surrounding window gave large corner response ( f > threshold) 3) Take the points of local maxima, i.e., perform non-maximum suppression Harris Detector: Steps Harris Detector: Steps Compute corner response f Find points with large corner response: f > threshold Harris Detector: Steps Harris Detector: Steps Take only the points of local maxima of f 6

CS 376: Computer Vision - lecture 12 2/27/2018 Properties of the Harris corner detector Properties of the Harris corner detector Rotation invariant? Yes Rotation invariant? Yes    0  1 T M X X    0   2 Scale invariant? Scale invariant? No All points will be Corner ! classified as edges Automatic Scale Selection Scale invariant interest points How can we independently select interest points in each image, such that the detections are repeatable across different scales? How to find corresponding patch sizes, with only one image in hand? K. Grauman, B. Leibe Automatic Scale Selection Automatic scale selection Intuition: • Function responses for increasing scale (scale signature) • Find scale that gives local maxima of some function f in both position and scale. f f Image 1 Image 2   f ( I ( x , )) f ( I ( x ,  )) i i i  i  1 m 1 m K. Grauman, B. Leibe s 1 region size s 2 region size 7

CS 376: Computer Vision - lecture 12 2/27/2018 Automatic Scale Selection Automatic Scale Selection • Function responses for increasing scale (scale signature) • Function responses for increasing scale (scale signature)     f ( I ( x , )) f ( I ( x , )) f ( I ( x ,  )) f ( I ( x ,  )) i  i i  i i  i i  i 1 m 1 m 1 m 1 m K. Grauman, B. Leibe K. Grauman, B. Leibe Automatic Scale Selection Automatic Scale Selection • Function responses for increasing scale (scale signature) • Function responses for increasing scale (scale signature)     f ( I ( x , )) f ( I ( x , )) f ( I ( x ,  )) f ( I ( x ,  )) i  i i  i i  i i  i 1 m 1 m 1 m 1 m K. Grauman, B. Leibe K. Grauman, B. Leibe Automatic Scale Selection What can be the “signature” function? • Function responses for increasing scale (scale signature)    f ( I ( x , )) f ( I ( x ,  )) i  i i  i 1 m 1 m K. Grauman, B. Leibe 8