Need for Regression Need for Linear Regression The Least Squares . . . Need to Go Beyond . . . Why Geometric Progression LASSO Method in Selecting the LASSO How λ Is Selected: . . . Natural Uniqueness . . . Parameter: A Theoretical Definitions and the . . . Discussion Explanation Home Page Title Page William Kubin 1 , Yi Xie 1 , Laxman Bokati 1 , Vladik Kreinovich 1 , and Kittawit Autchariyapanitkul 2 ◭◭ ◮◮ 1 Computational Science Program ◭ ◮ University of Texas at El Paso ElPaso, Texas 79968, USA Page 1 of 31 wkubin@miners.utep.edu, yxie3@miners.utep.edu, lbokati@miners.utep.edu, vladik@utep.edu Go Back 2 Maejo University, Thailand, kittawit a@mju.ac.th Full Screen Close Quit
Need for Regression Need for Linear Regression 1. Need for Regression The Least Squares . . . • In many real-life situations: Need to Go Beyond . . . LASSO Method – we know that the quantity y is uniquely determined How λ Is Selected: . . . by the quantities x 1 , . . . , x n , but Natural Uniqueness . . . – we do not know the exact formula for this depen- Definitions and the . . . dence. Discussion • For example, in physics: Home Page – we know that the aerodynamic resistance increases Title Page with the body’s velocity, but ◭◭ ◮◮ – we often do not know how exactly. ◭ ◮ • In economics: Page 2 of 31 – we know that a change in tax rate influences the Go Back economic growth, but Full Screen – we often do not know how exactly. Close Quit
Need for Regression Need for Linear Regression 2. Need for Regression (cont-d) The Least Squares . . . • In all such cases, we need to find the dependence y = Need to Go Beyond . . . f ( x 1 , . . . , x n ) between several quantities. LASSO Method How λ Is Selected: . . . • This dependence must be determined based on the Natural Uniqueness . . . available data. Definitions and the . . . • We need to use previous observations ( x k 1 , . . . , x kn , y k ) Discussion in each of which we know both: Home Page – the values x ki of the input quantities x i and Title Page – the value y k of the output quantity y . ◭◭ ◮◮ • In statistics, determining the dependence from the data ◭ ◮ is known as regression . Page 3 of 31 Go Back Full Screen Close Quit
Need for Regression Need for Linear Regression 3. Need for Linear Regression The Least Squares . . . • In most cases, the desired dependence is smooth – and Need to Go Beyond . . . usually, it can even be expanded in Taylor series. LASSO Method How λ Is Selected: . . . • In many practical situations, the range of the input variables is small, i.e., we have x i ≈ x (0) for some x (0) Natural Uniqueness . . . i . i Definitions and the . . . • In such situations, after we expand the desired depen- Discussion dence in Taylor series, we can: Home Page – safely ignore terms which are quadratic or of higher Title Page order with respect to the differences x i − x (0) and i ◭◭ ◮◮ – only keep terms which are linear in terms of these ◭ ◮ differences: n Page 4 of 31 � � � x i − x (0) y = f ( x 1 , . . . , x n ) = c 0 + a i · . i Go Back i =1 Full Screen = ∂f � � def x (0) 1 , . . . , x (0) def • Here c 0 = f and a i . n ∂x i | x i = x (0) Close i Quit
Need for Regression Need for Linear Regression 4. Need for Linear Regression (cont-d) The Least Squares . . . • This expression can be simplified into: Need to Go Beyond . . . LASSO Method n n def a i · x (0) � � y = a 0 + a i · x i , where a 0 = c 0 − i . How λ Is Selected: . . . i =1 i =1 Natural Uniqueness . . . • In practice, measurements are never absolutely precise. Definitions and the . . . Discussion • So, when we plug in the actually measured values x ki Home Page and y i , we will only get an approximate equality: Title Page m � y k ≈ a 0 + a i · x ki . ◭◭ ◮◮ i =1 ◭ ◮ • Thus, the problem of finding the desired dependence Page 5 of 31 can be reformulated as follows: Go Back – given the values y k and x ki , Full Screen – find the coefficients a i for which the approximate equality holds for all k . Close Quit
Need for Regression Need for Linear Regression 5. The Usual Least Squares Approach The Least Squares . . . • We want each left-and side y k of the approximate equal- Need to Go Beyond . . . ity to be close to the corresponding right-hand side. LASSO Method How λ Is Selected: . . . • In other words, we want the left-hand-side tuple ( y 1 , . . . , y K ) Natural Uniqueness . . . to be close to the right-hand-sides tuple � m Definitions and the . . . m � � � a i · x 1 i , . . . , a i · x Ki . Discussion Home Page i =1 i =1 Title Page • It is reasonable to select a i for which the distance be- tween these two tuples is the smallest possible. ◭◭ ◮◮ • Minimizing the distance is equivalent to minimizing the ◭ ◮ square of this distance, i.e., the expression Page 6 of 31 �� 2 K � � m Go Back � � y k − a 0 + a i · x ki . Full Screen k =1 i =1 • This minimization is know as the Least Squares method . Close Quit
Need for Regression Need for Linear Regression 6. The Least Squares Approach (cont-d) The Least Squares . . . • This is the most widely used method for processing Need to Go Beyond . . . data. LASSO Method How λ Is Selected: . . . • The corresponding values a i can be easily found if: Natural Uniqueness . . . – we differentiate the quadratic expression with re- Definitions and the . . . spect to each of the unknowns a i and then Discussion – equate the corresponding linear expressions to 0. Home Page • Then, we get an easy-to-solve systems of linear equa- Title Page tions. ◭◭ ◮◮ ◭ ◮ Page 7 of 31 Go Back Full Screen Close Quit
Need for Regression Need for Linear Regression 7. Discussion The Least Squares . . . • The above heuristic idea becomes well-justified: Need to Go Beyond . . . LASSO Method – when we consider the case when the measurement How λ Is Selected: . . . errors are normally distributed Natural Uniqueness . . . – with 0 mean and the same standard deviation σ . Definitions and the . . . • This indeed happens: Discussion Home Page – when the measuring instrument’s bias has been care- fully eliminated, and Title Page – most major sources of measurement errors have ◭◭ ◮◮ been removed. ◭ ◮ • In such situations, the resulting measurement error is a Page 8 of 31 joint effect of many similarly small error components. Go Back • For such joint effects, the Central Limit Theorem states Full Screen that the resulting distribution is close to Gaussian. Close Quit
Need for Regression Need for Linear Regression 8. Discussion (cont-d) The Least Squares . . . • Once we know the probability distributions, a natural Need to Go Beyond . . . idea is to select the most probable values a i . LASSO Method How λ Is Selected: . . . • In other words, we select the values for which the prob- Natural Uniqueness . . . ability to observe the values y k is the largest. Definitions and the . . . • For normal distributions, this idea leads exactly to the Discussion least squares method. Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 9 of 31 Go Back Full Screen Close Quit
Need for Regression Need for Linear Regression 9. Need to Go Beyond Least Squares The Least Squares . . . • Sometimes, we know that all the inputs x i are essential Need to Go Beyond . . . to predict the value y of the desired quantity. LASSO Method How λ Is Selected: . . . • In such cases, the least squares method works reason- Natural Uniqueness . . . ably well. Definitions and the . . . • The problem is that in practice, we often do not know Discussion which inputs x i are relevant and which are not. Home Page • As a result, to be on the safe side, we include as many Title Page inputs as possible. ◭◭ ◮◮ • Many of them will turn out to be irrelevant. ◭ ◮ • If all the measurements were exact, this would not be Page 10 of 31 a problem: Go Back – for irrelevant inputs x i , we would get a i = 0, and Full Screen – the resulting formula would be the desired one. Close Quit
Need for Regression Need for Linear Regression 10. Need to Go Beyond Least Squares (cont-d) The Least Squares . . . • However, because of the measurement errors, we do Need to Go Beyond . . . not get exactly 0s. LASSO Method How λ Is Selected: . . . • Moreover, the more such irrelevant variables we add: Natural Uniqueness . . . – the more non-zero “noise” terms a i · x i we will have, Definitions and the . . . and Discussion – the larger will be their sum. Home Page • This will negatively affecting the accuracy of the for- Title Page mula, ◭◭ ◮◮ • Thus, it will negative affect the accuracy of the result- ◭ ◮ ing desired (non-zero) coefficients a i . Page 11 of 31 Go Back Full Screen Close Quit
Recommend
More recommend
