Gov 2000: 10. Multiple Regression in Matrix Form Matthew Blackwell - PowerPoint PPT Presentation

Gov 2000: 10. Multiple Regression in Matrix Form Matthew Blackwell Fall 2016 1 / 64

1. Matrix algebra review 2. Matrix Operations 3. Linear model in matrix form 4. OLS in matrix form 5. OLS inference in matrix form 2 / 64

Where are we? Where are we going? and two independent variables, varying efgects covariates 3 / 64 • Last few weeks: regression estimation and inference with one • This week: the general regression model with arbitrary • Next week: what happens when assumptions are wrong

Nunn & Wantchekon Africans today? difgerent levels of historical slave exports for a respondent’s ethnic group interpersonal trust in ways that correlate with the severity of slave exports groups via multiple regression 4 / 64 • Are there long-term, persistent efgects of slave trade on • Basic idea: compare levels of interpersonal trust ( 𝑍 𝑗 ) across • Problem: ethnic groups and respondents might difger in their • One solution: try to control for relevant difgerences between

Nunn & Wantchekon Bold letter, quotation marks, what is this? 5 / 64 • Whaaaaa? • Today’s goal is to decipher this type of writing

Multiple Regression in R ## --- ## urban_dum -0.2738719 0.0143549 -19.08 <2e-16 *** ## malaria_ecology 0.0194106 0.0008712 22.28 <2e-16 *** ## Signif. codes: 2.01 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.978 on 20319 degrees of freedom ## (1497 observations deleted due to missingness) ## Multiple R-squared: 0.0604, Adjusted R-squared: 0.0602 ## F-statistic: 261 on 5 and 20319 DF, p-value: <2e-16 0.044 * 0.0138163 nunn <- foreign::read.dta("../data/Nunn_Wantchekon_AER_2011.dta") 0.0278369 summary(mod) ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 1.5030370 0.0218325 68.84 <2e-16 *** ## exports -0.0010208 0.0000409 -24.94 <2e-16 *** ## age 0.0050447 0.0004724 10.68 <2e-16 *** ## male 6 / 64 mod <- lm(trust_neighbors ~ exports + age + male + urban_dum + malaria_ecology, data = nunn)

Why matrices and vectors? 7 / 64

8 / 64

9 / 64 Why matrices and vectors? • Here’s one way to write the full multiple regression model: 𝑧 𝑗 = 𝛾 0 + 𝑦 𝑗1 𝛾 1 + 𝑦 𝑗2 𝛾 2 + ⋯ + 𝑦 𝑗𝑙 𝛾 𝑙 + 𝑣 𝑗 • Notation is going to get needlessly messy as we add variables. • Matrices are clean, but they are like a foreign language. • You need to build intuitions over a long period of time.

Quick note about interpretation conditional on all other 𝑦 𝑗𝑘 . “conditional on the covariates,” etc 10 / 64 𝑧 𝑗 = 𝛾 0 + 𝑦 𝑗1 𝛾 1 + 𝑦 𝑗2 𝛾 2 + ⋯ + 𝑦 𝑗𝑙 𝛾 𝑙 + 𝑣 𝑗 • In this model, 𝛾 1 is the efgect of a one-unit change in 𝑦 𝑗1 • Jargon “partial efgect,” “ceteris paribus,” “all else equal,” • Notation change: lower-case letters here are random variables.

1/ Matrix algebra review 11 / 64

Vectors ⎣ vectors will be written with lowercase bold lettering ( 𝐜 ) ⎦ ⎥ ⎥ ⎥ ⎤ 𝑏 𝑙 ⋮ 𝑏 2 𝑏 1 ⎢ ⎢ ⎢ ⎡ 𝐛 = ⋯ 𝑐 3 𝑐 2 12 / 64 • A vector is just list of numbers (or random variables). • A 1 × 𝑙 row vector has these numbers arranged in a row: 𝐜 = [ 𝑐 1 𝑐 𝑙 ] • A 𝑙 × 1 column vector arranges the numbers in a column: • Convention we’ll assume that a vector is column vector and

Vector examples exports 𝑗 ⎡ ⎢ ⎢ ⎢ ⎣ 1 age 𝑗 ⎥ male 𝑗 ⎤ ⎥ ⎥ ⎥ ⎦ ⎦ ⎥ ⎥ 1 ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝑦 𝑗1 𝑦 𝑗2 ⋮ 𝑦 𝑗𝑙 ⎤ ⎥ ⎥ 13 / 64 • Vector of all covariates for a particular unit 𝑗 : 𝐲 𝑗 = • For the Nunn-Wantchekon data, we might have: 𝐲 𝑗 =

Matrices 𝑏 2𝑙 ⎦ ⎥ ⎥ ⎥ ⎤ 𝑏 𝑜𝑙 ⋯ 𝑏 𝑜2 𝑏 𝑜1 ⋮ ⋱ ⋮ ⋮ ⋯ 𝑏 22 ⎢ columns. 𝐁 = ⎡ ⎢ ⎢ ⎣ 𝑏 11 𝑏 12 ⋯ 𝑏 1𝑙 𝑏 21 14 / 64 • A matrix is just a rectangular array of numbers. • We say that a matrix is 𝑜 × 𝑙 (“ 𝑜 by 𝑙 ”) if it has 𝑜 rows and 𝑙 • Uppercase bold denotes a matrix: • Generic entry: 𝑏 𝑗𝑘 where this is the entry in row 𝑗 and column 𝑘

Examples of matrices age 2 ⎦ ⎥ ⎥ ⎥ ⎤ male 𝑜 age 𝑜 exports 𝑜 1 ⋮ ⋮ ⋮ ⋮ male 2 exports 2 1 male 1 age 1 exports 1 1 ⎣ ⎢ ⎢ ⎢ ⎡ 𝐘 = regression. subsequent columns is each independent variable in the matrix, which has a column of ones, and then each of the 15 / 64 • One example of a matrix that we’ll use a lot is the design

Design matrix in R 855 31 1 0 28.15 ## 6 1 855 45 0 0 28.15 ## 7 1 20 1 1 0 28.15 ## 8 1 855 31 0 0 28.15 dim(model.matrix(mod)) ## [1] 20325 6 855 ## 5 head(model.matrix(mod), 8) 1 ## (Intercept) exports age male urban_dum malaria_ecology ## 1 1 855 40 0 0 28.15 ## 2 1 855 25 0 28.15 28.15 ## 3 1 855 38 1 1 28.15 ## 4 1 855 37 0 1 16 / 64

2/ Matrix Operations 17 / 64

Transpose 𝑏 32 ] 𝑏 32 𝑏 22 𝑏 12 𝑏 31 𝑏 21 ⎦ ⎥ ⎥ ⎤ 𝑏 31 𝑏 22 𝑏 21 𝑏 12 𝑏 11 ⎣ ⎢ ⎢ switching the rows and columns of the data and is denoted 𝐁 ′ . 18 / 64 • The transpose of a matrix 𝐁 is the matrix created by • 𝑙 th column of 𝐁 becomes the 𝑙 th row of 𝐁 ′ : 𝐁 = ⎡ 𝐁 ′ = [ 𝑏 11 • If 𝐁 is 𝑜 × 𝑙 , then 𝐁 ′ will be 𝑙 × 𝑜 . • Also written 𝐁 𝐔

Transposing vectors ⋮ ⋯ 𝑦 𝑗2 𝑦 𝑗1 𝐲 ′ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎤ 𝑦 𝑗𝑙 𝑦 𝑗2 𝑦 𝑗1 1 ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎡ vector and vice versa: 19 / 64 • Transposing will turn a 𝑙 × 1 column vector into a 1 × 𝑙 row 𝐲 𝑗 = 𝑗 = [ 1 𝑦 𝑗𝑙 ]

Transposing in R ## 6 5 ## [3,] 4 3 ## [2,] 2 1 ## [1,] [,1] [,2] t(a) a <- matrix(1:6, ncol = 3, nrow = 2) 6 4 2 ## [2,] 5 3 1 ## [1,] [,1] [,2] [,3] ## a 20 / 64

Write matrices as vectors 𝑏 22 𝑗 . ⎦ ⎥ ⎥ ⎤ 𝑐 32 𝑐 31 𝑐 22 𝑐 21 𝑐 12 𝑐 11 ⎣ ⎢ ⎢ 21 / 64 𝑏 12 𝑏 22 𝑏 12 𝑏 13 𝐛 ′ with row vectors ] 2 𝐛 ′ 1 𝑏 21 𝑏 23 • A matrix is just a collection of vectors (row or column) • As a row vector: 𝐁 = [ 𝑏 11 ] = [ 𝐛 ′ 1 = [ 𝑏 11 𝑏 13 ] 𝐛 ′ 2 = [ 𝑏 21 𝑏 23 ] • Or we can defjne it in terms of column vectors: 𝐂 = ⎡ = [ 𝐜 𝟐 𝐜 𝟑 ] where 𝐜 𝟐 and 𝐜 𝟑 represent the columns of 𝐂 . • 𝑘 subscripts columns of a matrix: 𝐲 𝑘 • 𝑗 and 𝑢 will be used for rows 𝐲 ′

Design matrix 2 = ⎡ ⎢ ⎢ ⎢ ⎣ 𝐲 ′ 1 𝐲 ′ ⋮ ⎥ 𝐲 ′ 𝑜 ⎤ ⎥ ⎥ ⎥ ⎦ 𝐲 1 𝐲 2 ⋯ ⎦ ⎥ ⎥ ⎤ 𝐘 = ⎡ ⎢ ⎢ ⎢ ⎣ 1 exports 1 age 1 male 1 1 exports 2 age 2 male 2 ⋮ ⋮ ⋮ ⋮ 1 exports 𝑜 age 𝑜 male 𝑜 22 / 64 • Design matrix as a series of row vectors: • Design matrix as a series of column vectors: 𝐘 = [ 𝟐 𝐲 𝑙 ]

Addition and subtraction 𝑐 12 = 𝐃 ] 𝑑 22 𝑑 21 𝑑 12 ] ] 𝑐 21 𝑐 22 𝑏 22 𝑏 21 𝑏 12 where we add each cell together: that the dimensions have to be the same. 23 / 64 • How do we add or subtract matrices and vectors? • First, the matrices/vectors need to be comformable, meaning • Let 𝐁 and 𝐂 both be 2 × 2 matrices. Then, let 𝐃 = 𝐁 + 𝐂 , 𝐁 + 𝐂 = [ 𝑏 11 ] + [ 𝑐 11 = [ 𝑏 11 + 𝑐 11 𝑏 12 + 𝑐 12 𝑏 21 + 𝑐 21 𝑏 22 + 𝑐 22 = [ 𝑑 11

Scalar multiplication like a 1 by 1 matrix. element/cell by that scalar: 𝑏 12 𝑏 21 𝑏 22 𝑐 × 𝑏 12 𝑐 × 𝑏 21 𝑐 × 𝑏 22 ] 24 / 64 • A scalar is just a single number: you can think of it sort of • When we multiply a scalar by a matrix, we just multiply each 𝑐𝐁 = 𝑐 [ 𝑏 11 ] = [ 𝑐 × 𝑏 11

3/ Linear model in matrix form 25 / 64

The linear model with new notation all 𝑗 ∈ {1, … , 𝑜} : (unit 1) (unit 2) (unit 3) (unit 4) 26 / 64 • Remember that we wrote the linear model as the following for 𝑧 𝑗 = 𝛾 0 + 𝑦 𝑗 𝛾 1 + 𝑨 𝑗 𝛾 2 + 𝑣 𝑗 • Imagine we had an 𝑜 of 4. We could write out each formula: 𝑧 1 = 𝛾 0 + 𝑦 1 𝛾 1 + 𝑨 1 𝛾 2 + 𝑣 1 𝑧 2 = 𝛾 0 + 𝑦 2 𝛾 1 + 𝑨 2 𝛾 2 + 𝑣 2 𝑧 3 = 𝛾 0 + 𝑦 3 𝛾 1 + 𝑨 3 𝛾 2 + 𝑣 3 𝑧 4 = 𝛾 0 + 𝑦 4 𝛾 1 + 𝑨 4 𝛾 2 + 𝑣 4

The linear model with new ⎦ 𝑨 2 𝑨 1 ⎣ ⎢ ⎢ ⎢ ⎡ ⎥ 𝑨 4 ⎥ ⎥ ⎤ 𝑦 4 𝑦 3 𝑦 2 𝑦 1 ⎣ 𝑨 3 ⎤ notation 𝑣 2 ⎦ ⎥ ⎥ ⎥ ⎤ 𝑣 4 𝑣 3 𝑣 1 ⎥ ⎣ ⎢ ⎢ ⎢ ⎡ ⎦ ⎥ ⎥ ⎢ ⎢ ⎢ ⎣ ⎡ ⎤ 𝑧 4 𝑧 3 𝑧 2 𝑧 1 ⎢ ⎥ ⎢ ⎢ ⎡ (unit 4) (unit 3) (unit 2) (unit 1) ⎥ ⎥ ⎦ 1 ⎦ ⎥ ⎥ ⎥ = 1 1 ⎤ 1 ⎣ ⎢ ⎢ ⎢ ⎡ 27 / 64 𝑧 1 = 𝛾 0 + 𝑦 1 𝛾 1 + 𝑨 1 𝛾 2 + 𝑣 1 𝑧 2 = 𝛾 0 + 𝑦 2 𝛾 1 + 𝑨 2 𝛾 2 + 𝑣 2 𝑧 3 = 𝛾 0 + 𝑦 3 𝛾 1 + 𝑨 3 𝛾 2 + 𝑣 3 𝑧 4 = 𝛾 0 + 𝑦 4 𝛾 1 + 𝑨 4 𝛾 2 + 𝑣 4 • We can write this as: 𝛾 0 + 𝛾 1 + 𝛾 2 + • Outcome is a linear combination of the the 𝐲 , 𝐴 , and 𝐯 vectors