Shape Matching Shape-Based Recognition Intro Humans can recognize - PowerPoint PPT Presentation

Shape Matching

Shape-Based Recognition Intro • Humans can recognize many objects based on shape alone • Fundamental cue for many object categories • Invariant to photometric variation. Slide from Pedro Felzenszwalb

Shapes vs. Intensity Values Intro Similar to a human in terms of shape, but very different in terms of pixel values. Images from Belongie et al.

Applications Intro • Shape retrieval • Recognizing object categories • Fingerprint identification • Optical Character Recognition (OCR) • Molecular-biology 1909 Western

Geometric Transformations Intro • Often in matching images are allowed to undergo some geometric transformation • Related but not identical shapes can be deformed into alignment using simple coordinate transformations • Find the transformations of one image that produce good matches to the other image Images from Belongie et al.

Biological Shape Intro • D’Arcy Thompson: On Growth and Form , 1917 • studied transformations between shapes of organisms d c b a 0 1 2 3 4 5 Fig. 177. Human skull Fig. 179. Skull of chimpanzee. Fig. 180. Skull of baboon. Slide from Belongie et al.

Related Problems Intro • Shape representation and decomposition • Finding a set of correspondences between shapes • Transforming one shape into another • Measuring the similarity between shapes • Shape localization and model alignment • Finding a shape similar to a model in a cluttered image ¼ Slide from Pedro Felzenszwalb

References Intro • Shape Matching and Object Recognition Using Shape Contexts , by S. Belongie, J. Malik, and J. Puzicha. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2002. • Recognizing Objects in Adversarial Clutter: Breaking a Visual CAPTCHA , by G. Mori and J. Malik, in Proceedings IEEE Computer Vision and Pattern Recognition (CVPR), 2003. • Using the Inner-Distance for Classification of Articulated Shapes , by H. Ling and D. Jacobs, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005. • Comparing Images Using the Hausdorff Distance , by D. Huttenlocher, G. Klanderman, and W. Rucklidge, Transactions on Pattern Analysis and Machine Intelligence (PAMI), 1993. • A Boundary-Fragment-Model for Object Detection , by A. Opelt, A. Pinz, and A. Zisserman, Proceedings of the European Conference on Computer Vision (ECCV), 2006. • Hierarchical Matching of Deformable Shapes , by P. Felzenszwalb and J. Schwartz, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2007

Hausdorff Outline Distance • Shape Distance and Correspondence ➢ Hausdorff Distance • Shape Context • Inner Distance • Hierarchical Approach • Hierarchical Matching • Machine Learning Approach • Boundary Fragment Model

Hausdorff Distance Comparing Images Using the Hausdorff Distance 1993 D. Huttenlocher, G. Klanderman, and W. Rucklidge

Hausdorff Overview Distance • Use Hausdorff distance to compare images to a model • Fast and simple approach • Tolerant of small position errors • Model is only allowed to translate with respect to the image • Can be extended to allow rotation and scale

Hausdorff Hausdorff Distance Distance • A means of determining the resemblance of one point set to another • Examines the fraction of points in one set that lie near points in the other set H ( A; B ) = max f h ( A; B ) ; h ( B; A ) g ½ ¾ b 2 B f d ( a; b ) g h ( A; B ) = max min a 2 A

Hausdorff Example Distance a 2 a 1 Given two sets of points A and B, find h(A,B) b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 Compute the distance between a 1 and each b j b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 Keep the shortest b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 Do the same for a 2 b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 Find the largest of these two distances b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 This is h(A,B) b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 This is h(B,A) b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 H(A,B) = max(h(A,B),h(B,A)) b 3 b 1 b 2

Hausdorff Example Distance a 2 a 1 This is H(A,B) b 3 b 1 b 2

Hausdorff Generalization Distance • Hausdorff distance is very sensitive to even one outlier in A or B • Use k th ranked distance instead of the maximal distance • Match if h k ( A; B ) < ± • is how many points of the model need to be k near points of the image • is how near these points need to be ± ½ ¾ h k ( A; B ) = k th b 2 B f d ( a; b ) g min a 2 A

Hausdorff Distance Transforms Distance • Processing can be sped up by probing a precomputed Voronoi surface • A Voronoi surface defines the distance from any location in B to the nearest point • Can be efficiently computed using dynamic programming in linear time

Hausdorff Example: Matching Distance Model Edges Match

Hausdorff Example: Matching Distance Model Model Image Edges Match

Shape Outline Context • Shape Distance and Correspondence • Hausdorff Distance ➢ Shape Context • Inner Distance • Hierarchical Approach • Hierarchical Matching • Machine Learning Approach • Boundary Fragment Model

Shape Context Shape Matching and Object Recognition Using Shape Contexts 2002 S. Belongie, J. Malik, and J. Puzicha

Shape Overview Context 1) Solve for correspondences between points on the two shapes ● Using shape contexts 2) Use the correspondences to estimate an aligning transform ● Using regularized thin-plate splines 3) Compute the distance between the two shapes

Shape Related Work: Deformable Templates Context • The Representation and Matching of Pictorial Structures , by Fischler & Elschlager (1973) • Structural image restoration through deformable templates , by Grenander et al. (1991) • Deformable Templates for Face Recognition , by Yuille (1991) • Distortion invariant object recognition in the dynamic linkarchitecture , by von der Malsburg (1993) Slide from Belongie et al.

Shape Sampling Points Context • A shape is represented by a set of points sampled from the edges of the object

Shape Shape Context: Log-Polar Histograms Context Count the number of points Count = 4 inside each bin. Count = 12 Slide from Belongie et al.

Shape Example: Shape Contexts Context a) b) c) d) Images from Belongie et al.

Shape Point Correspondences Context • Compute matching costs using C ( p i ; p j ) Chi Squared distance: X K [ h i ( k ) ¡ h j ( k )] 2 C ( p i ; p j ) = 1 2 h i ( k ) + h j ( k ) k =1 • Minimize the total cost of matching, such that matching is 1-to-1 X C ¡ p i ; q ¼ ( i ) ¢ H ( ¼ ) = i [Jonker & Volgenant, 1987] Slide from Belongie et al.

Shape Example: Point Correspondences Context a) b) c)

Shape Thin Plate Spline Model Context • The name “thin plate spline” refers to a physical analogy involving the bending of a thin sheet of metal • The 2D generalization of the 1D cubic spline • Contains the affine model as a special case

Shape Minimizing Bend Energy Context • The Thin Plate Spline interpolation has the form: X n w i U ( jj ( x i ; y i ) ¡ ( x; y ) jj ) f ( x; y ) = a 1 + a x x + a y y + i =1 | {z } | {z } global affine transform local non-linear transformations where, U ( r ) = r 2 log r 2 • Select and to minimize the bend a w energy: Z Z µ @ 2 f ¶ 2 µ @ 2 f ¶ 2 µ @ 2 f ¶ 2 I ( f ) = + 2 + 2 dxdy @x 2 @y 2 @x@y R 2

Shape Example: Matching and Transformation Context a) b) Images from Belongie et al.

Shape Terms in Similarity Score Context • Shape Context difference, D sc • Local Image appearance difference, D ac • Orientation • Gray-level correlation in Gaussian window • … (many more possible) • Bending energy, D be D sc + 1 : 6 ¤ D ac + 0 : 3 ¤ D be

Shape Shape Context Results Context Query Similarity Scores 0.086 0.108 0.109 0.066 0.073 0.077 0.046 0.107 0.114 0.117 0.121 0.129 0.096 0.147 0.153 Images from Belongie et al.

Inner Outline Distance • Shape Distance and Correspondence • Hausdorff Distance • Shape Context ➢ Inner Distance • Hierarchical Approach • Hierarchical Matching • Machine Learning Approach • Boundary Fragment Model

Inner Distance Using the Inner-Distance for Classification of Articulated Shapes 2005 H. Ling and D. Jacobs

Inner Overview Distance • Its difficult to capture the part structure of complex shapes with existing shape matching methods • Replace euclidean distance with the inner- distance • Insensitive to shape articulations • Often more discriminative for complex shapes • An extension to shape contexts

Inner Model of Articulated Objects Distance 1) An object can be decomposed into a number of parts 2) Junctions between parts are relatively small with respect to the parts they connect 3) Articulation on the object is rigid with respect to any part, but can be non-rigid on the junctions 4) An object that has been articulated can be articulated back to its original form Images from Ling and Jacobs