Incorporating Detractors into SVM Marcin Orchel AGH University of - PowerPoint PPT Presentation

Agenda Support Vector Machines Generalized SVC Reduced Models Incorporating Detractors into SVM Marcin Orchel AGH University of Science and Technology Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models 1 Support Vector Machines 2 Generalized SVC 3 Reduced Models Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models Support Vector Machines Support Vector Machines (SVM) – a set of supervised learning methods used mainly for classification and regression problems SVM maximal margin classifier – classifies data with the hyperplane that has the largest distance to the closest training vectors SVM soft margin classifier - extended version able to classify nonseparable data H ξ 1 � � w � ξ 2 � � w � Marcin Orchel Incorporating Detractors into SVM Figure:

Agenda Support Vector Machines Generalized SVC Reduced Models Support Vector Classification (SVC) Formulation SVC primal problem: minimization of = 1 w � 2 + � � � w , b , � C · � f � 2 � � ξ ξ a i ) ≥ 1 − ξ i , � with constraints: y i h ( � ξ ≥ 0 for i ∈ { 1 .. l } , where � C ≫ 0, h ( � a i ) = � w · � a i + b . SVC dual problem: maximization of α − 1 � W ( � α ) = 1 · � α T Q � α 2 with constraints � α · � y = 0 , 0 ≤ � α ≤ C where Q ij = y i y j ( � a i · � a j ) for all i , j ∈ { 1 . . . l } . Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models SVC Solution decision boundary l h ∗ ( � � c α ∗ x ) + b ∗ y i x ) = i K ( � c = 0 (1) a i ,� i =1 non support vectors have α ∗ i = 0 they are ignored in the solution H ξ 1 � w � ξ 2 � w � Marcin Orchel Incorporating Detractors into SVM Figure:

Agenda Support Vector Machines Generalized SVC Reduced Models Generalized SVC Idea How to make non support vectors useful and include them in the solution? generalized SVC – add possibility for points outside margins to be support vectors H ξ 1 ξ 2 � w � � w � Marcin Orchel Incorporating Detractors into SVM Figure:

Agenda Support Vector Machines Generalized SVC Reduced Models Generalized SVC Formulation Use generalized SVC formulation: minimization of = 1 � � w � 2 + � w , b , � C · � f � ξ 2 � � ξ a i ) ≥ 1 − ξ i + ϕ i , � with constraints: y i h ( � ξ ≥ 0 for i ∈ { 1 .. l } , where � C ≫ 0, ϕ i ∈ R , h ( � a i ) = � w · � a i + b . Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models Generalized SVC – Interpretation of the Weights H ϕ 4 � w � ξ 1 � w � ϕ 3 ξ 2 � w � � w � Figure: Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models Reduced Models How to use these new weights? Proposed reduced method: 1 run the SVC on the original data 2 compute values of the new weights based on the found solution 3 reduce the data by removing some of support vectors 4 run generalized SVC on the reduced data with values of the new weights the proposed method preserves additional knowledge about the original solution in the new weights Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models Properties of Reduced Models reduced models are models with increased sparsity, reduced models are faster for postprocessing (because of simpler solutions), e.g. testing new examples reduction ratio parameter is the trade-off between generalization performance and simplicity to some extent reducing data with the proposed method does not decrease generalization performance result: we are able to generate reduced models with similar number of support vectors and better generalization performance than for the simple reduction Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models Results increased generalization performance of reduced models ranging from 5% to 70% comparing to the simple reduction with similar number of support vectors 0 0 -20 -20 -40 -40 -60 -60 -80 -80 -100 -100 -120 -120 10 20 30 40 50 60 70 10 20 30 40 50 60 70 Figure: Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models Conclusions with the generalized SVC we can create simpler SVC solutions – with less support vectors even without compromising generalization performance reduced models are faster in post-processing the research is financed by the Polish Ministry of Science and Higher Education project No NN519579338 I would like to express my sincere gratitude to Professor Witold Dzwinel (AGH University of Science and Technology, Department of Computer Science). Questions Marcin Orchel Incorporating Detractors into SVM