least squares support vector regression with applications
play

Least Squares Support Vector Regression with Applications to - PowerPoint PPT Presentation

Feb 24, 2024 •1.28k likes •2.28k views

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least Squares Support Vector Regression with Applications to Large-Scale Data: a Statistical Approach Kris De

  1. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least Squares Support Vector Regression with Applications to Large-Scale Data: a Statistical Approach Kris De Brabanter Public Defense April, 27 2011 Promotor: Prof. dr. ir. B. De Moor Co-Promotor: Prof. dr. ir. J. Suykens 1/39

  2. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Outline Goal & Overview 1 Introduction 2 Parametric vs. nonparametric regression Nonparametric regression estimates: an overview Fixed-Size Least Squares Support Vector Machines 3 Fixed Size LS-SVM formulation Selection of Support Vectors Practical identification problem Robust Nonparametric Methods 4 Problems with outliers Robust nonparametric regression Correlated Errors 5 Problems with correlation in nonparametric regression Removing correlation effects Confidence Intervals 6 Conclusions 7 2/39

  3. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Outline Goal & Overview 1 Introduction 2 Parametric vs. nonparametric regression Nonparametric regression estimates: an overview Fixed-Size Least Squares Support Vector Machines 3 Fixed Size LS-SVM formulation Selection of Support Vectors Practical identification problem Robust Nonparametric Methods 4 Problems with outliers Robust nonparametric regression Correlated Errors 5 Problems with correlation in nonparametric regression Removing correlation effects Confidence Intervals 6 Conclusions 7 3/39

  4. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Goal of the Thesis Goal of the Thesis Study the properties of Least Squares Support Vector Machines for regression with an emphasis on statistical aspects and develop a framework for large scale data 4/39

  5. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Overview Model Selection Large Scale (Chapter 3) Data Sets (Chapter 4) Model Building (Chapter 2) Robustness (Chapter 5) Least Squares Introduction Support Vector (Chapter 1) Machines Correlated Errors (Chapter 6) Conclusions & Further Research Confidence (Chapter 9) Intervals Applications & (Chapter 7) Case Studies (Chapter 8) 5/39

  6. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Outline Goal & Overview 1 Introduction 2 Parametric vs. nonparametric regression Nonparametric regression estimates: an overview Fixed-Size Least Squares Support Vector Machines 3 Fixed Size LS-SVM formulation Selection of Support Vectors Practical identification problem Robust Nonparametric Methods 4 Problems with outliers Robust nonparametric regression Correlated Errors 5 Problems with correlation in nonparametric regression Removing correlation effects Confidence Intervals 6 Conclusions 7 6/39

  7. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 20 10 0 −10 −20 −30 −5 0 5 7/39

  8. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 20 10 0 −10 −20 −30 −5 0 5 7/39

  9. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 20 Y = aX + b 10 0 −10 −20 −30 −5 0 5 7/39

  10. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 7/39

  11. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 7/39

  12. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = ? Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 7/39

  13. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = ? Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 PARAMETRIC FORM IS NOT ALWAYS EASY TO FIND 7/39

  14. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 8/39

  15. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 8/39

  16. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x 8/39

  17. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x 8/39

  18. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 m n ( x ) ˆ 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x 8/39

  19. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 m n ( x ) ˆ 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x n K ( x − X i ) � h m n ( x ) = ˆ Y i j =1 K ( x − X j � n ) i =1 h 8/39

  20. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Other nonparametric regression estimates Local constant regression (Nadaraya, 1964; Watson, 1964) Regression trees (Breiman et al. , 1984) Wavelets (Daubechies, 1992) Nearest Neighbors (Devroye et al. , 1994) Local linear regression (Fan & Gijbels, 1996) Support vector machines (Vapnik, 1995) Splines (Wahba, 1990; Eubank, 1999) Partitioning estimates (Gy¨ orfi et al. , 2002) Least squares support vector machines (Suykens et al. , 2002) . . . 9/39

  21. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ w,b,e J P ( w, e ) = 1 � n k =1 e 2 min k 2 w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. 10/39

  22. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ � n w,b,e J P ( w, e ) = 1 k =1 e 2 min 2 k w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. input space ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 10/39

  23. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ � n w,b,e J P ( w, e ) = 1 k =1 e 2 min 2 k w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. input space ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 10/39

  24. Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ � n w,b,e J P ( w, e ) = 1 k =1 e 2 min 2 k w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. input space feature space ⋆ ϕ ( · ) ⋆ ϕ ( ) ⋆ ⋆ ϕ ( ) ⋆ ⋆ ⋆ ϕ ( ) ϕ ( ) ⋆ ϕ ( ) ⋆ ⋆ ϕ ( ) ⋆ ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ⋆ ⋆ ⋆ ϕ ( ) ϕ ( ) ⋆ ⋆ ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 10/39

Recommend

On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine

On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine

Index On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine Regression Characterization of vectors in SVM regression Procedure for Adding one vector Procedure for Removing one vector

520 views • 15 slides
What is a What are Support Vector Machines Support Vector Machine? Used For? An optimally

What is a What are Support Vector Machines Support Vector Machine? Used For? An optimally

What is a What are Support Vector Machines Support Vector Machine? Used For? An optimally defined surface Classification Typically nonlinear in the input space Regression and data-fitting Linear in a higher dimensional

577 views • 8 slides
Regression Based on Support Vector Classification Marcin Orchel AGH University of Science and

Regression Based on Support Vector Classification Marcin Orchel AGH University of Science and

Agenda Support Vector Classification and Regression The New Regression Method Regression Based on Support Vector Classification Marcin Orchel AGH University of Science and Technology Marcin Orchel Regression Based on Support Vector

480 views • 11 slides
1 Least Squares Regression Suppose someone hands you a stack of N vectors, { x N } , each of

1 Least Squares Regression Suppose someone hands you a stack of N vectors, { x N } , each of

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 3B notes: Least Squares Regression 1 Least Squares Regression Suppose someone hands you a stack of N vectors, { x N } , each

291 views • 4 slides
Support Vector Regression with a Priori Knowledge Used in Order Execution Strategies Based on

Support Vector Regression with a Priori Knowledge Used in Order Execution Strategies Based on

Agenda Order Execution Support Vector Machines Support Vector Regression with a Priori Knowledge Used in Order Execution Strategies Based on VWAP Marcin Orchel AGH University of Science and Technology in Poland Supervisor: prof. Witold

303 views • 11 slides
Least Mean Squares Regression Machine Learning 1 Least Squares Method for regression

Least Mean Squares Regression Machine Learning 1 Least Squares Method for regression

Least Mean Squares Regression Machine Learning 1 Least Squares Method for regression Examples The LMS objective Gradient descent Incremental/stochastic gradient descent 2 Least Squares Method for regression Examples The

1.23k views • 53 slides
Support Vector Machines 290N, 2014 Support Vector Machines (SVM) Supervised learning

Support Vector Machines 290N, 2014 Support Vector Machines (SVM) Supervised learning

Support Vector Machines 290N, 2014 Support Vector Machines (SVM) Supervised learning methods for classification and regression they can represent non-linear functions and they have an efficient training algorithm derived

1.15k views • 57 slides
MACHINE LEARNING Linear and Weighted Regression Support Vector Regression 1 APPLIED MACHINE

MACHINE LEARNING Linear and Weighted Regression Support Vector Regression 1 APPLIED MACHINE

APPLIED MACHINE LEARNING MACHINE LEARNING Linear and Weighted Regression Support Vector Regression 1 APPLIED MACHINE LEARNING Classification (reminder) Maps N -dimensions input x N to discrete values y E.g.: x = [ Length , Color ]

931 views • 68 slides
FORECASTING DIRECTION OF CHINA SECURITY INDEX 300 MOVEMENT WITH LEAST SQUARES SUPPORT VECTOR

FORECASTING DIRECTION OF CHINA SECURITY INDEX 300 MOVEMENT WITH LEAST SQUARES SUPPORT VECTOR

FORECASTING DIRECTION OF CHINA SECURITY INDEX 300 MOVEMENT WITH LEAST SQUARES SUPPORT VECTOR MACHINE Shuai Wang ,Wei Shang Academy of Mathematics and Systems Sciences Chinese Academy of Sciences Beijing, China , June 2014 Contents 1

356 views • 20 slides
Chapter 5 Least Squares Chapter 5 Inconsistent Systems In regression (and many other

Chapter 5 Least Squares Chapter 5 Inconsistent Systems In regression (and many other

Chapter 5 Least Squares Chapter 5 Inconsistent Systems In regression (and many other applications) we have a system of equations wed like to solve: X = y However, this system does not have an exact solution. (i.e. all of our data points

386 views • 11 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Least Squares for Classification K-class

467 views • 24 slides
Least Squares (outline) Standard regression: Fit data with weighted sum of regressors.

Least Squares (outline) Standard regression: Fit data with weighted sum of regressors.

Mathematical Tools for Neural and Cognitive Science Fall semester, 2018 Section 2: Least Squares Least Squares (outline) Standard regression: Fit data with weighted sum of regressors. Solution via calculus, orthogonality, SVD

377 views • 9 slides
Kernel Methods for Regression Support Vector Regression Gaussian Mixture Regression Gaussian

Kernel Methods for Regression Support Vector Regression Gaussian Mixture Regression Gaussian

MACHINE LEARNING MACHINE LEARNING MACHINE LEARNING Kernel Methods for Regression Support Vector Regression Gaussian Mixture Regression Gaussian Process Regression 1 MACHINE LEARNING 2012 MACHINE LEARNING MACHINE LEARNING Problem

945 views • 71 slides
Kernel Methods and Support Vector Machines Oliver Schulte - CMPT 726 Bishop PRML Ch. 6 Support

Kernel Methods and Support Vector Machines Oliver Schulte - CMPT 726 Bishop PRML Ch. 6 Support

Kernel Methods and Support Vector Machines Oliver Schulte - CMPT 726 Bishop PRML Ch. 6 Support Vector Machines Defining Characteristics Like logistic regression, good for continuous input features, discrete target variable. Like

646 views • 19 slides
Support Vector Machines and their Applications Purushottam Kar Department of Computer Science and

Support Vector Machines and their Applications Purushottam Kar Department of Computer Science and

Introduction SVMs Generalization Bounds The Kernel Trick Implementations Applications Support Vector Machines and their Applications Purushottam Kar Department of Computer Science and Engineering, Indian Institute of Technology Kanpur.

1.07k views • 73 slides
Weighted Least Squares Recall the linear regression equation E ( Y ) = 0 + 1 x 1 + 2 x 2

Weighted Least Squares Recall the linear regression equation E ( Y ) = 0 + 1 x 1 + 2 x 2

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II Weighted Least Squares Recall the linear regression equation E ( Y ) = 0 + 1 x 1 + 2 x 2 + + k x k We have estimated the

680 views • 11 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Linear Regression Solution to Maximum

145 views • 11 slides
Support Vector Manifold Learning for Solving Regression Problems via Clustering Marcin Orchel

Support Vector Manifold Learning for Solving Regression Problems via Clustering Marcin Orchel

Support Vector Manifold Learning for Solving Regression Problems via Clustering Marcin Orchel AGH University of Science and Technology in Poland 1 / 29 2 / 29 1.0 y -1.0 1.0 x (a) 3 / 29 new kernel c )) T ( ( ( ( x i ) +

878 views • 29 slides
? 17.10.2018 3 17.10.2018 4 Support Vector Machines (SVM): Background Support Vector Machines

? 17.10.2018 3 17.10.2018 4 Support Vector Machines (SVM): Background Support Vector Machines

INF3490 - Biologically inspired computing Support Vector Machines Support Vector Machines, Ensemble Learning, and Dimensionality (SVM) Reduction Weria Khaksar October 17, 2018 17.10.2018 2 Support Vector Machines (SVM): Background Support

579 views • 9 slides
Support Vector Machines Preview What is a support vector machine? The perceptron revisited

Support Vector Machines Preview What is a support vector machine? The perceptron revisited

Support Vector Machines Preview What is a support vector machine? The perceptron revisited Kernels Weight optimization Handling noisy data What Is a Support Vector Machine? 1. A subset of the training examples x (the support

971 views • 19 slides
Business Statistics CONTENTS Ordinary least squares (recap for some) Statistical formulation of

Business Statistics CONTENTS Ordinary least squares (recap for some) Statistical formulation of

SIMPLE REGRESSION ANALYSIS Business Statistics CONTENTS Ordinary least squares (recap for some) Statistical formulation of the regression model Assessing the regression model Testing the regression coefficients The ANOVA table Old exam

555 views • 30 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Logistic Regression 2- class logistic

308 views • 15 slides
Experiments in Value Function Approximation with Sparse Support Vector Regression Tobias Jung and

Experiments in Value Function Approximation with Sparse Support Vector Regression Tobias Jung and

Experiments in Value Function Approximation with Sparse Support Vector Regression Tobias Jung and Thomas Uthmann { tjung,uthmann } @informatik.uni-mainz.de Fachbereich Mathematik & Informatik Johannes Gutenberg-Universit at Mainz, Germany

438 views • 17 slides
8. Least squares Review of linear equations Least squares Example: curve-fitting

8. Least squares Review of linear equations Least squares Example: curve-fitting

CS/ECE/ISyE 524 Introduction to Optimization Spring 201718 8. Least squares Review of linear equations Least squares Example: curve-fitting Vector norms Geometrical intuition Laurent Lessard (www.laurentlessard.com)

453 views • 23 slides

More recommend