Data Dependence in Data Dependence in Combining Classifiers Combining Classifiers Mohamed Kamel, Nayer Wanas Mohamed Kamel, Nayer Wanas Pattern Analysis and Machine Intelligence Lab Pattern Analysis and Machine Intelligence Lab University of Waterloo University of Waterloo CANADA CANADA
Outline Outline Introduction Introduction Data Dependence Data Dependence ! Implicit Dependence Implicit Dependence ! ! Explicit Dependence Explicit Dependence ! Feature Based Architecture Feature Based Architecture ! Training Algorithm Training Algorithm ! Results Results Conclusions Conclusions
Introduction Introduction Pattern Recognition Systems Pattern Recognition Systems Introduction Introduction Data Dependence Implicit ! Best possible classification rates. Best possible classification rates. ! Explicit Feature Based ! Increase efficiency and accuracy. Increase efficiency and accuracy. Training ! Results Conclusions Multiple Classifier Systems Multiple Classifier Systems ! Evidence of improving performance Evidence of improving performance ! ! Problem decomposed naturally from using Problem decomposed naturally from using ! various sensors various sensors ! Avoid making commitments to arbitrary initial Avoid making commitments to arbitrary initial ! conditions or parameters conditions or parameters MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS Categorization of MCS Architecture Architecture Introduction Introduction Data Dependence Input/Output Mapping Implicit Input/Output Mapping Explicit Feature Based Representation Representation Training Results Conclusions Types of classifiers Types of classifiers MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS ( Categorization of MCS (cntd cntd…) …) Architecture Architecture Parallel [ Parallel [Dasarathy Dasarathy, 94] , 94] Introduction Introduction Data Dependence Implicit Input 1 F Classifier 1 Explicit Feature Based U Output Input 2 Training S Classifier 2 Results I Conclusions O Input N N Classifier N Serial [ Serial [Dasarathy Dasarathy, 94] , 94] Output Input 1 Classifier 1 Classifier 2 Classifier N Input 2 Input N MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS ( Categorization of MCS (cntd cntd…) …) Architectures [Lam, 00] Architectures [Lam, 00] Conditional Topology Conditional Topology Introduction Introduction Data Dependence ! Once a classifier unable to classify the output the Once a classifier unable to classify the output the ! Implicit Explicit following classifier is deployed following classifier is deployed Feature Based Training Hierarchal Topology Hierarchal Topology Results Conclusions ! Classifiers applied in succession Classifiers applied in succession ! ! Classifiers with various levels of generalization Classifiers with various levels of generalization ! Hybrid Topology Hybrid Topology ! The choice of the classifier to use is based on the The choice of the classifier to use is based on the ! input pattern (selection) input pattern (selection) Multiple (Parallel) Topology Multiple (Parallel) Topology MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS ( Categorization of MCS (cntd cntd…) …) Input/Output Mapping Input/Output Mapping Linear Mapping Linear Mapping Introduction Introduction Data Dependence ! Sum Rule Sum Rule Implicit ! Explicit ! Weighted Average Weighted Average [ [Hashem Hashem 97] 97] Feature Based ! Training Non- -linear Mapping linear Mapping Results Non Conclusions ! Maximum Maximum ! ! Product Product ! ! Hierarchal Mixture of Experts Hierarchal Mixture of Experts [Jordon and Jacobs 94] [Jordon and Jacobs 94] ! ! Stacked Generalization Stacked Generalization [ [Wolpert Wolpert 92] 92] ! MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS ( Categorization of MCS (cntd cntd…) …) Representation Representation Similar representations Similar representations Introduction Introduction Data Dependence ! Classifiers need to be different Classifiers need to be different Implicit ! Explicit Different representation Feature Based Different representation Training Results ! Use of different sensors Use of different sensors ! Conclusions ! Different features extracted from the same data set Different features extracted from the same data set ! [Ho, 98, Skurichina Skurichina & & Duin Duin, 02] , 02] [Ho, 98, MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS ( Categorization of MCS (cntd cntd…) …) Types of Classifiers Types of Classifiers Specialized classifiers Specialized classifiers Introduction Introduction Data Dependence ! Encourage specialization in areas of the feature space Encourage specialization in areas of the feature space Implicit ! Explicit ! All classifiers must contribute to achieve a final decision All classifiers must contribute to achieve a final decision Feature Based ! Training ! Hierarchal Mixture of Experts Hierarchal Mixture of Experts [Jordon and Jacobs 94] [Jordon and Jacobs 94] Results ! Conclusions ! Co Co- -operative Modular Neural Networks operative Modular Neural Networks [ [Auda Auda and Kamel 98] and Kamel 98] ! Ensemble of classifiers Ensemble of classifiers ! Set of redundant classifiers Set of redundant classifiers ! Competitive versus cooperative [Sharkey, 1999] Competitive versus cooperative [Sharkey, 1999] MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Categorization of MCS ( Categorization of MCS (cntd cntd…) …) Data Dependence Data Dependence Introduction Introduction Data Dependence ! Classifiers inherently dependent on the data. Classifiers inherently dependent on the data. Implicit ! Explicit Feature Based ! Describe how the final aggregation uses the Describe how the final aggregation uses the ! Training Results information present in the input pattern. information present in the input pattern. Conclusions ! Describe the relationship between the final Describe the relationship between the final ! output Q(x) Q(x) and the pattern under and the pattern under output classification x x classification MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Data Dependence Data Dependence Data Independent Data Independent Introduction Data Dependence Data Dependence Implicitly Dependent Implicit Implicitly Dependent Explicit Feature Based Explicitly Dependent Explicitly Dependent Training Results Conclusions MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Data Independence Data Independence Solely rely on output of classifiers to determine Solely rely on output of classifiers to determine Introduction Data Dependence final classification output. Data Dependence final classification output. Implicit Explicit = ∀ Feature Based Q(x) arg max (F (C (x)), j) Training j j Results j Conclusions Q(x) is the final class assigned for pattern Q(x) is the final class assigned for pattern x x C j is a vector composed of the output of the various C j is a vector composed of the output of the various classifiers in the ensemble { c c 1j ,c 2j ,...,c c Nj } for a given for a given classifiers in the ensemble { 1j ,c 2j ,..., Nj } class y y j class j c ij is the confidence classifier i i has in pattern has in pattern x x c ij is the confidence classifier belonging to class y y j belonging to class j Mapping F F j can be linear or non- -linear linear Mapping j can be linear or non MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Data Independence ( Data Independence (cntd cntd…) …) Simple voting techniques are data independent Simple voting techniques are data independent Introduction Data Dependence Data Dependence ! Average Average ! Implicit Explicit ! Maximum Maximum ! Feature Based Training ! Majority Majority ! Results Conclusions Susceptible to incorrect estimates of the confidence Susceptible to incorrect estimates of the confidence MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Implicit Data Dependence Implicit Data Dependence Train the combiner on global performance of the Train the combiner on global performance of the Introduction Data Dependence data data Implicit Implicit Explicit Feature Based = ∀ Training ( ( )), Q(x) arg max (F (W C x C (x)), j) Results j j Conclusions j W(C (x)) is the weighting matrix composed of W(C (x)) is the weighting matrix composed of elements w elements w ij ij w ij is the weight assigned to class j j in classifier in classifier i i w ij is the weight assigned to class MCS 2003 MCS 2003 Data Dependence in Combining Classifiers Data Dependence in Combining Classifiers
Recommend
More recommend
