Additional Topics on Linear Regression Ping Yu School of Economics - PowerPoint PPT Presentation

Additional Topics on Linear Regression Ping Yu School of Economics and Finance The University of Hong Kong Ping Yu (HKU) Additional Topics 1 / 49

Tests for Functional Form Misspecification 1 Nonlinear Least Squares 2 Omitted and Irrelevant Variables 3 Model Selection 4 Generalized Least Squares 5 Testing for Heteroskedasticity 6 7 Regression Intervals and Forecast Intervals Ping Yu (HKU) Additional Topics 2 / 49

Review of Our Assumptions Assumption OLS.0 (random sampling): ( y i , x i ) , i = 1 , ��� , n , are i.i.d. Assumption OLS.1 (full rank): rank ( X ) = k . Assumption OLS.1 0 : rank ( E [ xx 0 ]) = k . Assumption OLS.2 (first moment): E [ y j x ] = x 0 β . Assumption OLS.2 0 : y = x 0 β + u with E [ x u ] = 0 . Assumption OLS.3 (second moment): E [ u 2 ] < ∞ . Assumption OLS.3 0 (homoskedasticity): E [ u 2 j x ] = σ 2 . Assumption OLS.4 (normality): u j x � N ( 0 , σ 2 ) . h k x k 4 i Assumption OLS.5 : E [ u 4 ] < ∞ and E < ∞ . Ping Yu (HKU) Additional Topics 2 / 49

y = x 0 β + u Implied Properties E [ x u ] = 0 linear projection consistency = ) [ E [ u j x ] = 0 linear regression = ) unbiasedness [ E [ u j x ] = 0 homoskedastic = ) Gauss-Markov Theorem E [ u 2 j x ] = σ 2 linear regression Table 1: Relationship between Different Models Ping Yu (HKU) Additional Topics 3 / 49

Our Plan of This Chapter Examine the validity of these assumptions and cures when they fail, especially, OLS.2 and OLS.3 0 . Assumption OLS.2 has many implications. For example, it implies - the conditional mean of y given x is linear in x . - all relevant regressors are included in x and are fixed. Section 1 and 2 examine the first implication: - Section 1 tests whether E [ y j x ] is indeed x 0 β . - Section 2 provides more flexible specifications of E [ y j x ] . Section 3 and 4 examine the second implication: - Section 3 checks the benefits and costs of including irrelevant variables or omitting relevant variables. - Section 4 provides some model selection procedures based on information criteria. Section 5 and 6 examine Assumption OLS.3 0 : - Section 5 shows that there are more efficient estimators of β when this assumption fails. - Section 6 checks whether this assumption fails. Section 7 examines the external validity of the model. Ping Yu (HKU) Additional Topics 4 / 49

Tests for Functional Form Misspecification Tests for Functional Form Misspecification Ping Yu (HKU) Additional Topics 5 / 49

Tests for Functional Form Misspecification Ramsey’s REgression Specification Error Test (RESET) Misspecification of E [ y j x ] may be due to omitted variables or misspecified functional forms. We only examine the second source of misspecification here. A Straightforward Test: add nonlinear functions of the regressors to the regression, and test their significance using a Wald test. i e - fit y i = x 0 β + z 0 i e γ + e u i by OLS, and form a Wald statistic for γ = 0 , where z i = h ( x i ) denote functions of x i which are not linear functions of x i (perhaps squares of non-binary regressors). RESET: The null model is y i = x 0 i β + u i . Let 0 1 y 2 b i B C . z i = . @ A . y m b i i b y i = x 0 be an ( m � 1 ) -vector of powers of b β . Then run the auxiliary regression i e y i = x 0 β + z 0 i e γ + e u i (1) by OLS, and form the Wald statistic W n for γ = 0 . Ping Yu (HKU) Additional Topics 6 / 49

Tests for Functional Form Misspecification continue... d ! χ 2 Under H 0 , W n � m � 1 . Thus the null is rejected at the α level if W n exceeds the upper α tail critical value of the χ 2 m � 1 distribution. To implement the test, m must be selected in advance. Typically, small values such as m = 2 , 3 , or 4 seem to work best. The RESET test is particularly powerful in detecting the single-index model, y i = G ( x 0 β ) + u i , where G ( � ) is a smooth "link" function. Why? Note that (1) may be written as � � 2 e � � m e i e i b i b y i = x 0 x 0 x 0 γ m � 1 + e β + β γ 1 + ��� + β u i , which has essentially approximated G ( � ) by an m th order polynomial. Other tests: Hausman test, Conditional Moment (CM) test, Zheng’s score test. Ping Yu (HKU) Additional Topics 7 / 49

Nonlinear Least Squares Nonlinear Least Squares Ping Yu (HKU) Additional Topics 8 / 49

Nonlinear Least Squares Nonlinear Least Squares If the specification test rejects the linear specification, we may consider to use a nonlinear setup for E [ y j x ] . Suppose E [ y i j x i = x ] = m ( x j θ ) . Nonlinear regression means that m ( x j θ ) is a nonlinear function of θ (rather than x ). The functional form of m ( x j θ ) can be suggested by an economic model, or as in the LSE, it can be treated as a nonlinear approximation to a general conditional mean function. m ( x j θ ) = exp ( x 0 θ ) : Exponential Link Regression - The exponential link function is strictly positive, so this choice can be useful when it is desired to constrain the mean to be strictly positive. m ( x j θ ) = θ 1 + θ 2 x θ 3 , x > 0: Power Transformed Regressors - A generalized version of the power transformation is the famous Box-Cox (1964) transformation, where the regressor x θ 3 is generalized as x ( θ 3 ) with ( x λ � 1 if λ > 0 , , x ( λ ) = λ if λ = 0 . log x , - The function x ( λ ) nests linearity ( λ = 1) and logarithmic ( λ = 0) transformations continuously. Ping Yu (HKU) Additional Topics 9 / 49

Nonlinear Least Squares 1.5 1 0.5 0 -0.5 -1 -1.5 -2 -2.5 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Figure: Box-Cox Transformation for Different λ Values: all pass ( 1 , 0 ) Ping Yu (HKU) Additional Topics 10 / 49

Nonlinear Least Squares continue... m ( x j θ ) = θ 1 + θ 2 exp ( θ 3 x ) : Exponentially Transformed Regressors m ( x j θ ) = G ( x 0 θ ) , G known - When G ( � ) = ( 1 + exp ( �� )) � 1 , the regression with m ( x j θ ) = G ( x 0 θ ) is called the logistic link regression . - When G ( � ) = Φ ( � ) with Φ ( � ) being the cdf of standard normal, it is called the probit link regression . � � x 2 � θ 3 m ( x j θ ) = θ 0 1 x 1 + θ 0 2 x 1 G : Smooth Transition θ 4 m ( x j θ ) = θ 1 + θ 2 x + θ 3 ( x � θ 4 ) 1 ( x > θ 4 ) : Continuous Threshold Regression m ( x j θ ) = ( θ 0 1 x 1 ) 1 ( x 2 � θ 3 ) + ( θ 0 2 x 1 ) 1 ( x 2 > θ 3 ) : Threshold Regression - When θ 4 = 0, the smooth transition model (STM) reduces to the threshold regression (TR) model. - When θ 4 = ∞ , the STM reduces to linear regression. - For CTR and TR, m ( x j θ ) is not smooth in θ . Ping Yu (HKU) Additional Topics 11 / 49

Nonlinear Least Squares 2 2 1.5 1.5 1 1 0.5 0.5 0 0 0 0.5 1 1.5 2 0 0.5 1 1.5 2 2 2 1.5 1.5 1 1 0.5 0.5 0 0 0 0.5 1 1.5 2 0 0.5 1 1.5 2 Figure: Difference Between STM, CTR and TR Ping Yu (HKU) Additional Topics 12 / 49

Nonlinear Least Squares The NLLS Estimator The nonlinear least squares (NLLS) estimator is n ∑ b ( y i � m ( x i j θ )) 2 . θ = argmin θ i = 1 The FOCs for minimization are 0 = ∑ n i = 1 m θ ( x i j b u i with m θ ( x j θ ) = ∂ θ ) b ∂θ m ( x j θ ) . Theorem If the model is identified and m ( x j θ ) is differentiable with respect to θ , � � p d b n θ � θ 0 ! N ( 0 , V ) , � h i � � � 1 E � � � 1 with m θ i = m θ ( x i j θ 0 ) . m θ i m 0 m θ i m 0 θ i u 2 m θ i m 0 where V = E E θ i i θ i V can be estimated by ! � 1 ! � 1 ! � 1 n n n 1 1 1 ∑ ∑ ∑ b m 0 m 0 u 2 m 0 m θ i b b m θ i b b θ i b m θ i b b V = , θ i i θ i n n n i = 1 i = 1 i = 1 m θ i = m θ ( x i j b u i = y i � m ( x i j b where b θ ) , and b θ ) . Ping Yu (HKU) Additional Topics 13 / 49

Omitted and Irrelevant Variables Omitted and Irrelevant Variables Ping Yu (HKU) Additional Topics 14 / 49

Omitted and Irrelevant Variables Omitted Variables The true model is a long regression y i = x 0 1 i β 1 + x 0 2 i β 2 + u i , E [ x i u i ] = 0 , (2) but we estimate a short regression y i = x 0 1 i γ 1 + v i , E [ x 1 i v i ] = 0 . (3) β 1 6 = γ 1 . Why? � � � 1 E [ x 1 i y i ] x 1 i x 0 γ 1 = E 1 i � � � 1 E [ x 1 i � � ] x 1 i x 0 x 0 1 i β 1 + x 0 = E 2 i β 2 + u i 1 i � � � 1 E [ x 1 i x 0 x 1 i x 0 = β 1 + E 2 i ] β 2 1 i = β 1 + Γ β 2 , � � � 1 � 1 E [ x 1 i x 0 x 1 i x 0 where Γ = E 2 i ] is the coefficient from a regression of x 2 i on 1 i x 1 i . Ping Yu (HKU) Additional Topics 15 / 49

Omitted and Irrelevant Variables Direct and Indirect Effects of x 1 on y γ 1 6 = β 1 unless Γ = 0 or β 2 = 0 . The former means that the regression of x 2 i on x 1 i yields a set of zero coefficients (they are uncorrelated), and the latter means that the coefficient on x 2 i in (2) is zero. The difference Γ β 2 is known as omitted variable bias . γ 1 includes both the direct effect of x 1 on y ( β 1 ) and the indirect effect ( Γ β 2 ) through x 2 . [Figure here] To avoid omitted variable bias the standard advice is to include potentially relevant variables in the estimated model. But many desired variables are not available in a given dataset. In this case, the possibility of omitted variable bias should be acknowledged and discussed in the course of an empirical investigation. Ping Yu (HKU) Additional Topics 16 / 49

Omitted and Irrelevant Variables Figure: Direct and Indirect Effects of x 1 on y Ping Yu (HKU) Additional Topics 17 / 49