Regression Based on Support Vector Classification Marcin Orchel - PowerPoint PPT Presentation

Agenda Support Vector Classification and Regression The New Regression Method Regression Based on Support Vector Classification Marcin Orchel AGH University of Science and Technology Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method 1 Support Vector Classification and Regression 2 The New Regression Method Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Support Vector Classification and Regression two the most popular problems in machine learning: classification and regression for both problems there were developed methods in the framework of Support Vector Machines (SVM): Support Vector Classification (SVC) and ε -insensitive Support Vector Regression ( ε -SVR) similarities between SVC and ε -SVR: quadratic optimization problems they lead to sparse solutions they originate from statistical learning theory for ǫ -SVR we have the additional parameter ε Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method ε -SVR Idea find a function for which all examples fall between ε bounds y ε ε Figure: Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Idea of the New Regression Method data transformation: regression examples are duplicated and transformed in the way that original examples are translated up and duplicated examples are translated down by some parameter ϕ ≥ 0 y y ϕ ϕ Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Idea of the New Regression Method 1 problem transformation: the output y of the regression problem is incorporated to the feature set as the additional feature 2 a classification problem is solved with the new data setting 3 the solution of the classification problem is transformed to the regression function y = . . . Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Transforming the Curve to the Function SVC can return nonlinear solutions using the kernel trick nonlinear classifier can lead to the solution which cannot be transformed to the function (more than one y value for some values of the remaining features) popular kernels: polynomial, sigmoid, radial basis functions (RBF) we propose a slightly modification of these kernels by skipping last variable which is summed separately, e.g. � m � d � d � m +1 � � + x m +1 y m +1 → x i y i x i y i i =1 i =1 Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Details of the New Regression How to choose the best value of ϕ ? we use a simple search method we have found experimentally that it is better to compare regression performance on training data than classification performance while choosing ϕ for symmetrical, unimodal distributions of regression data, the transformed classification problem has the same optimal solution as the original regression problem for any ϕ Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Advantages we need only the SVC solver for classification and regression problems modifications of SVC original problem such as adding a priori knowledge could be directly used for regression problems for example we can create improved reduced models for regression problems by using generalized SVC Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Results test performed on synthetic data with added Gaussian noise and on real world data sets with ϕ we can control the number of support vectors results show that the new regression method is able to achieve comparable to the ε -SVR or better generalization performance on unseen data Marcin Orchel Regression Based on Support Vector Classification

Agenda Support Vector Classification and Regression The New Regression Method Conclusions the new regression is an alternative for ǫ -SVR it is based on SVC, therefore it is easier to use some modifications of SVC like incorporating a priori knowledge directly for regression problems the research is financed by the Polish Ministry of Science and Higher Education project No NN519579338 I would like to express my sincere gratitude to Professor Witold Dzwinel (AGH University of Science and Technology, Department of Computer Science). Questions Marcin Orchel Regression Based on Support Vector Classification