SYDE 372 - Winter 2011 Introduction to Pattern Recognition Distance - PowerPoint PPT Presentation

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier SYDE 372 - Winter 2011 Introduction to Pattern Recognition Distance Measures for Pattern Classification: Part II Alexander Wong Department of Systems Design Engineering University of Waterloo Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Outline Weighted Euclidean Distance Metric 1 Orthonormal Covariance Transforms 2 Generalized Euclidean Metric 3 Minimum Intra-Class Distance (MICD) Classifier 4 Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Weighted Euclidean distance metric Motivation: problem with using Euclidean distance is that pattern space in general is NOT in Euclidean vector space! Different measurements and features may: be more or less dependent have different units and scales have different variances The use of Euclidean distance can lead to poor classification performance in certain cases where the above situations hold true. Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Example where Euclidean distance can cause issues The Euclidean distance from x to class mean prototype z 1 is shorter than that to cluster mean prototype z 2 , even though intuitively it should belong to class 2. Could use NN prototypes, but that is more computationally expensive and less robust to noise Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Weighted Euclidean distance metric Idea: Since the features may have different units, scales, and variances, why don’t we weight the features differently when measuring distances? � n � 1 2 1 � d W o ( x , z ) = ( w i ( x i − z i )) (1) 2 i = 1 What we are doing is essentially scaling the feature axes with a linear transformation and then applying Euclidean distance metric. d W o ( x , z ) = d E ( x ′ , z ′ ) (2) where x ′ = W o x , z ′ = W o x , and W o is a diagonal matrix of weights. Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Example revisited The weighted Euclidean distance in the original feature space is just Euclidean distance in transformed space! Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Example Suppose we are given the following statistical information about the classes: � 3 � 0 Class 1: m 1 = [ 3 3 ] T , S 1 = . 0 1 � 2 � 0 Class 2: m 2 = [ 4 5 ] T , S 2 = . 0 1 Suppose we wish to build a WED classifier using sample means as prototypes and the following weight matrices W 0 , 1 and W 0 , 2 : √ √ � � � � 1 / 3 0 1 / 2 0 W 0 , 1 = W 0 , 2 = (3) 0 1 0 1 Compute the discriminate function for each class. Compute the decision boundary. Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Example Step 1: Find discriminant functions for each class based on WED decision rule: Recall that the WED decision criteria for the two class case is: d W o ( x , z 1 ) < d W o ( x , z 2 ) (4) o , 1 W o , 1 ( x − z 1 )] 1 / 2 < [( x − z 2 ) T W T [( x − z 1 ) T W T o , 2 W o , 2 ( x − z 2 )] 1 / 2 (5) ( x − z 1 ) T W T o , 1 W o , 1 ( x − z 1 ) < ( x − z 2 ) T W T o , 2 W o , 2 ( x − z 2 ) (6) Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Example Plugging in z 1 = m 1 , z 2 = m 2 , W o , 1 , and W o , 2 gives us: ( x − z 1 ) T W T o , 1 W o , 1 ( x − z 1 ) < ( x − z 2 ) T W T o , 2 W o , 2 ( x − z 2 ) (7) � T � ([ x 1 x 2 ] T − [ 3 3 ] T ) T � � √ √ ([ x 1 x 2 ] T − [ 3 3 ] T ) 1 / 3 0 1 / 3 0 0 1 0 1 � T � < ([ x 1 x 2 ] T − [ 4 5 ] T ) T � � √ √ ([ x 1 x 2 ] T − [ 4 5 ] T ) 1 / 2 0 1 / 2 0 0 1 0 1 (8) � 1 / 3 � 0 ([ x 1 − 3 x 2 − 3 ]) T ([ x 1 − 3 x 2 − 3 ]) 0 1 � 1 / 2 (9) 0 � ([ x 1 − 4 x 2 − 5 ]) T < ([ x 1 − 4 x 2 − 5 ]) 0 1 Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Example Plugging in z 1 = m 1 , z 2 = m 2 , W o , 1 , and W o , 2 gives us: � 1 / 3 � 0 ([ x 1 − 3 x 2 − 3 ]) T ([ x 1 − 3 x 2 − 3 ]) 0 1 � 1 / 2 (10) � 0 ([ x 1 − 4 x 2 − 5 ]) T < ([ x 1 − 4 x 2 − 5 ]) 0 1 ([( x 1 − 3 ) / 3 x 2 − 3 ])([ x 1 − 3 x 2 − 3 ]) T (11) < ([( x 1 − 4 ) / 2 x 2 − 5 ])([ x 1 − 4 x 2 − 5 ]) T ( x 1 − 3 ) 2 / 3 + ( x 2 − 3 ) 2 < ( x 1 − 4 ) 2 / 2 + ( x 2 − 5 ) 2 (12) Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Example Expanding gives us: ( x 1 − 3 ) 2 / 3 + ( x 2 − 3 ) 2 < ( x 1 − 4 ) 2 / 2 + ( x 2 − 5 ) 2 (13) 2 x 2 1 + 6 x 2 2 − 12 x 1 − 36 x 2 + 72 < 3 x 2 1 + x 2 2 − 24 x 1 − 10 x 2 + 73 (14) Therefore, the discriminant functions are: g 1 ( x 1 , x 2 ) = 2 x 2 1 + 6 x 2 2 − 12 x 1 − 36 x 2 + 72 (15) g 2 ( x 1 , x 2 ) = 3 x 2 1 + x 2 2 − 24 x 1 − 10 x 2 + 73 (16) Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier MED Classifier: Decision Boundary Step 2: Find decision boundary between classes 1 and 2 For WED classifier, the decision boundary is g ( x 1 , x 2 ) = g 1 ( x 1 , x 2 ) − g 2 ( x 1 , x 2 ) = 0 . (17) Plugging in the discriminant functions g 1 and g 2 gives us: g ( x 1 , x 2 ) = 2 x 2 1 + 6 x 2 2 − 12 x 1 − 36 x 2 + 72 (18) − ( 3 x 2 1 + x 2 2 − 24 x 1 − 10 x 2 + 73 ) = 0 Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Decision Boundary Grouping terms: g ( x 1 , x 2 ) = − x 2 1 + 5 x 2 2 + 12 x 1 − 26 x 2 − 1 = 0 (19) Therefore, the decision boundary is a quadratic! Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Decision Boundary The decision boundary for a MED classifier looks like this Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier WED Classifier: Decision Boundary The decision boundary for this WED classifier looks like this Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Weighted Euclidean distance metric A more general form of the weighted Euclidean distance metric can be defined as: � 1 � ( x − z ) T W T W ( x − z ) 2 d W ( x , z ) = (20) where W is the general weight matrix of the form:   w 11 w 12 . . . w 1 n w 21 w 12   (21)  .  ... .   .   w n 1 w nn Allows scaling AND rotation of axes! Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Weighted Euclidean distance metric Question: Why do we care about rotation of the axes? Answer: Cases like this... Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Orthonormal Covariance Transforms Question: How do we determine the weights W ? Intuition: Euclidean distance is only valid for cases where features are: uncorrelated unit variance Visually, shape of distribution in feature space is a hypersphere. Therefore, we wish to find W that transforms the shape of the distribution into a hypersphere! Alexander Wong SYDE 372 - Winter 2011

Weighted Euclidean Distance Metric Orthonormal Covariance Transforms Generalized Euclidean Metric Minimum Intra-Class Distance (MICD) Classifier Desired Transform: Visualization Alexander Wong SYDE 372 - Winter 2011