Applied Statistics Lecturer: Serena Arima Regression model Model - PowerPoint PPT Presentation

Regression model Model estimation Properties OLS estimator Applied Statistics Lecturer: Serena Arima

Regression model Model estimation Properties OLS estimator Linear regression model Consider the income and the expenditure of a sample of n = 1000 italian individuals in 2010. Possible questions: 1 Are the income and the expenditure related? 2 How are these variables related? 3 Can we use these variables to predict the expenditure of the next year corresponding to a fixed income?

Regression model Model estimation Properties OLS estimator Linear regression model A linear regression model can be specified as follows y i = β 0 + β 1 x i 1 + β 2 x i 2 + ... + β k x ik + ǫ i = x ′ i β + ǫ i where y i is the response variable; x i 1 , .., x ik are the explanatory variables or predictors; ǫ i is the unobserved random term.

Regression model Model estimation Properties OLS estimator Linear regression model The equality y i = x ′ i β + ǫ i is supposed to hold for any possible observation, while we only observe a sample of n observations. We shall consider this sample as one realization of all potential samples of size n that have been drawn from the same population. In this way, we can view y i and ǫ i as random variables . In the regression context, we consider the predictors as observed and fixed (deterministic). Using the matrix notation, Y = X β + ǫ

Regression model Model estimation Properties OLS estimator Linear regression model Mathematical model ↔ Statistical model Change of measurement scale Statistical model y=100*x Height and weight measurements 10000 ● ● ● ● ● ● ● ● ● 200 ● ● 8000 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 150 ● ● ● ● ● 6000 ● ● ● ● ● Length in cm ● ● ● ● ● ● Height ● ● ● ● ● ● ● ● 100 ● ● ● 4000 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 50 ● ● ● ● ● ● ● 2000 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● 0 ● ● 0 20 40 60 80 100 0 20 40 60 80 100 Length in m Weight

Regression model Model estimation Properties OLS estimator Linear regression model Our goal is to find the coefficient of the linear combination β 0 , ˜ ˜ β 1 , ˜ β 2 ... ˜ β k that minimize the following objective function N � S (˜ ( y i − ˜ β 0 − ˜ β 1 x 1 i − ˜ β 2 x 2 i − ... − ˜ β k x ki ) 2 β ) = i = 1 N � i ˜ β ) 2 = ( y i − x ′ i = 1 That is we minimize the sum of squared approximation errors. This approach is referred to ordinary least squares or OLS approach .

Regression model Model estimation Properties OLS estimator Simple linear model Suppose we want to estimate a simple regression model y i = β 0 + β 1 x i + ǫ i that is we want to estimate the regression line y i = � β 0 + � � β 1 x i such that � n � n y i ) 2 = S ( � β 0 , � e 2 β 1 ) = ( y i − � i i = 1 i = 1 is minimum . y i � predicted values; � 1 2 e i � residuals.

Regression model Model estimation Properties OLS estimator Simple linear model Suppose we want to estimate a simple regression model y i = β 0 + β 1 x i + ǫ i that is we want to estimate the regression line y i = � β 0 + � � β 1 x i such that � n � n y i ) 2 = S ( � β 0 , � e 2 β 1 ) = ( y i − � i i = 1 i = 1 is minimum . y i � predicted values; � 1 2 e i � residuals.

Regression model Model estimation Properties OLS estimator Simple linear model We can estimate � β 0 and � β 1 with the OLS method. n dS ( � β 0 , � � β 1 ) ( y i − � β 0 − � = − 2 β 1 x i ) = 0 d β 0 i = 1 � n dS ( � β 0 , � β 1 ) x i ( y i − � β 0 − � = − 2 β 1 x i ) = 0 d β 1 i = 1

Regression model Model estimation Properties OLS estimator Simple linear model Solving the system we get: � y − � = ¯ β 1 ¯ β 0 x � n i = 1 ( x i − ¯ x )( y i − ¯ y ) � = � n = β 1 i = 1 ( x i − ¯ x ) � n i = 1 x i y i − n ¯ x ¯ y � n x 2 = i = 1 x 2 i − n ¯ Cov ( X , Y ) � = Var ( x ) � Var ( Y ) ρ Var ( X )

Regression model Model estimation Properties OLS estimator Example 1 Italian income and expenditure data 1 : we have selected a sample of 1000 of italians and we have collected the following variables: Income (annual income); Expenditure (annual expenditure); Age; Number of components in the family. 1 Data from Banca d’Italia

Regression model Model estimation Properties OLS estimator Example 1 We want to study the relationship between the expenditure and the income with the following linear model: Expenditure i = β 0 + β 1 Income i + ǫ i ( i = 1 , ..., 1000 ) Income and Expenditure (data on log scale) 12.0 ● 11.5 ● ● ● 11.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 10.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Expenditure ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 10.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 9.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 9.0 ● ● ● ● ● ● ● ● ● ● 8.5 ● ● 8 9 10 11 12 Income

Regression model Model estimation Properties OLS estimator Example 1 The regression line has been estimated in R and it is estimated as � Expenditure i = 2 . 8907 + 0 . 6947 Income i ( i = 1 , ..., 1000 ) How to interpret these coefficients? � β 0 :2.8907 is the average expenditure for a subject i with null income; � β 1 : increasing the income of 1 unit, the average expenditure increases of 0.6947. More formally The parameter β 1 measures the expected change in y i if x i changes with one unit.

Regression model Model estimation Properties OLS estimator Example 1 Suppose we want to predict the expenditure of a family with Income equal to 3. According to our model, the predicted value is 2 . 8907 + 0 . 6947 × 3 = 4 . 9748 And what is the predicted expenditure of a family with Income equal to 0?

Regression model Model estimation Properties OLS estimator Example 1 Suppose we want to predict the expenditure of a family with Income equal to 3. According to our model, the predicted value is 2 . 8907 + 0 . 6947 × 3 = 4 . 9748 And what is the predicted expenditure of a family with Income equal to 0?

Regression model Model estimation Properties OLS estimator Example 1 Suppose we want to predict the expenditure of a family with Income equal to 3. According to our model, the predicted value is 2 . 8907 + 0 . 6947 × 3 = 4 . 9748 And what is the predicted expenditure of a family with Income equal to 0?

Regression model Model estimation Properties OLS estimator Example 2: Wages data Wages data 2 : a sample ( n = 3294) of individual wages with background characteristics like gender, race and years of schooling. We want to study the relationship between the wages and gender. So we have: y 1 , ..., y 3294 : wages; x 1 , .., x 3294 : 0 − 1 variable denoting whether the individual is male ( x i = 1) or female ( x i = 0). � dummy variable 2 Chapter 2, Verbeek’s book