MACHINE LEARNING Linear and Weighted Regression Support Vector - PowerPoint PPT Presentation

APPLIED MACHINE LEARNING MACHINE LEARNING Linear and Weighted Regression Support Vector Regression 1

APPLIED MACHINE LEARNING Classification (reminder) Maps N -dimensions input x ∈ ℝ N to discrete values y E.g.: x = [ Length , Color ] “Banana” or “Apple” Length Apples Bananas Color How to estimate a continuous output value y? 2

APPLIED MACHINE LEARNING Regression: introduction Maps N-dimensions input x ∈ ℝ N to continuous values y ∈ ℝ Income (GDP) Continuous value of life satisfaction Italy 7 US Japan 6 Life satisfaction India Nigeria 5 China 4 Bangladesh Cambodia 3 0 10 000 20 000 30 000 40 000 Income: GDP 2003 (log scale) 3

APPLIED MACHINE LEARNING Regression: introduction Maps N-dimensions input x ∈ ℝ N to continuous values y ∈ ℝ Income (GDP) Continuous value of life satisfaction Estimation of life satisfaction = 6.5 Italy 7 US Japan 6 Life satisfaction India Nigeria 5 Query point: Russia China 4 GDP = 30 000 Bangladesh Cambodia 3 0 10 000 20 000 30 000 40 000 Income: GDP 2003 (log scale) 4

APPLIED MACHINE LEARNING Example of Use of Regressive Methods Predict the number of diplomas that will be awarded in the next ten years across the two EPF  the number of diploma follow a non-linear curve as a function of time. School of Engineering – Section Microtechnique @ 2004 A.. Billard – Adapted from Blei 99 and Dorr & Montz 2004 5

APPLIED MACHINE LEARNING Example of Use of Regressive Methods    x f x Predict the velocity of the robot given its position. x*: target 6

APPLIED MACHINE LEARNING Example of Use of Regressive Methods 7

APPLIED MACHINE LEARNING Linear Regression Linear regression searches a linear mapping between input x and output y, parametrized by the slope vector w and intercept b.      T ; , y f x w b w x b y b x 8

APPLIED MACHINE LEARNING Linear Regression Linear regression searches a linear mapping between input x and output y, parametrized by the slope vector w and intercept b.      T ; , y f x w b w x b One can omit the intercept by centering the data:     y ' y y and ' x x x , , x y : mean on and x y   T y ' w x ' b '    T with ' b b w x y   *    T Least-square estimate of b ' y ' w x ' 0   T y ' w x '. 9

APPLIED MACHINE LEARNING Linear Regression Linear regression searches a linear mapping between input x and output y, parametrized by the slope vector w.     T ; y f x w w x y x 10

APPLIED MACHINE LEARNING Linear Regression   M M Pair of M training points X [ x x ... x ] and y [ y y ... y ] 1 2 1 2   i N i ℝ ℝ , . x y Find the optimal parameter w through least-square regression :     M 1  2   * T w min  w x y  i i   2 w  i 1 Finds an analytical solution through partial differentiation:    1 * = T T w X X X Y 11

APPLIED MACHINE LEARNING Weighted Linear Regression Regression through weighted Least Square     M 1 2            ℝ * T i   w min w x y , & ... i i i 1 2 M   2 w  1 i  Standard linear regression y ˆ : estimator y x All points have equal weight. 12

APPLIED MACHINE LEARNING Weighted Linear Regression Regression through weighted Least Square     M 1 2       ℝ * T i   w min w x y , i i i   2 w  1 i y ˆ y x Points in red have large weights. 13

APPLIED MACHINE LEARNING Weighted Linear Regression Regression through weighted Least Square     M 1 2       ℝ * T i   w min w x y , i i i   2 w  1 i y ˆ y x Points in red have large weights. 14

APPLIED MACHINE LEARNING Weighted Linear Regression  Assuming a set of weights for all datapoints, we set B a diagonal matrix i    1         2 with entries , B  i .....      ................... M   T Change of variable: Z BX and v By . Minimizing f or MSE, one gets an estimator for at the query point: y    1   ˆ T T T T y x w x Z Z Z v Contrast to the solution for un-weighted linear regression    1  * T T w X X X y 15

APPLIED MACHINE LEARNING Limitations of Linear Regression Regression through weighted Least Square     M 1 2       ℝ * T i w min  w x y  , : constant weights i i i   2 w  i 1 assumes that a single linear dependency applies everywhere. Not true for data sets with local dependencies. y x 16

APPLIED MACHINE LEARNING Limitations of Linear Regression Regression through weighted Least Square     M 1 2       ℝ * T i w min  w x y  , : constant weights i i i   2 w  i 1 assumes that a single linear dependency applies everywhere. Not true for data sets with local dependencies.  It would be useful to design a regression method that estimates best the linear dependencies locally. 17

APPLIED MACHINE LEARNING Locally Weighted Regression Estimate is determined through local influence of each group of datapoints M M                ℝ y x x y / x x : weights function of x i i j i   i 1 j 1                i d x , x     i x = K d x x , , with K d x x , e , d x x , x x . i i i i y   ˆ y x X: query point 18

APPLIED MACHINE LEARNING Locally Weighted Regression Estimate is determined through local influence of each group of datapoints M M                ℝ y x x y / x x : weights function of x i i j i   i 1 j 1                i d x , x     i x = K d x x , , with K d x x , e , d x x , x x . i i i i y X: query point Generates a smooth   ˆ y x function y(x) 19

APPLIED MACHINE LEARNING Locally Weighted Regression Estimate is determined through local influence of each group of datapoints M M                ℝ y x x y / x x : weights function of x i i j i   i 1 j 1        i x = K d x x , i Model-free regression!  T y w x No longer explicit model of the form Regression computed at each query point. Depends on training points. 20

APPLIED MACHINE LEARNING Locally Weighted Regression Estimate is determined through local influence of each group of datapoints M M                ℝ y x x y / x x : weights function of x i i j i   i 1 j 1        i x = K d x x , i Optimal solution to the local cost function:   2     M      ˆ i min J x min y y K d x x , i  i 1 Local cost function at , the query point. x 21

APPLIED MACHINE LEARNING Locally Weighted Regression Estimate is determined through local influence of each group of datapoints M M                ℝ y x x y / x x : weights function of x i i j i   i 1 j 1 Which training points?        i x = K d x x , i Which kernel? 22

APPLIED MACHINE LEARNING Exercise Session Part I 23

APPLIED MACHINE LEARNING Data-driven Regression Good prediction depends on the choice of datapoints. y x Blue : true function Red : estimated function 24

APPLIED MACHINE LEARNING Data-driven Regression Good prediction depends on the choice of datapoints. The more datapoints, the better the fit. Computational costs increase dramatically with number of datapoints y x Blue : true function Red : estimated function 25

APPLIED MACHINE LEARNING Data-driven Regression Several methods in ML for performing non-linear regression. Differ in the objective function, in the amount of parameters. Gaussian Process Gaussian Process Regression (GPR) uses all datapoints (model-free) Regression not covered in class! Not examined in the final exam! y x Blue : true function Red : estimated function 26

APPLIED MACHINE LEARNING Data-driven Regression Several methods in ML for performing non-linear regression. Differ in the objective function, in the amount of parameters. Gaussian Process Regression (GPR) uses all datapoints (model-free) Support Vector Regression (SVR) picks a subset of datapoints (support vectors) y x Blue : true function Red : estimated function 27