PS 406 Week 1 Section: Review of OLS and Matrix Algebra D.J. Flynn - PowerPoint PPT Presentation

PS 406 – Week 1 Section: Review of OLS and Matrix Algebra D.J. Flynn April 4, 2014 D.J. Flynn PS406 – Week 1 Section Spring 2014 1 / 24

Today’s plan Logistics 1 Matrix algebra review 2 OLS review 3 Lab 1 4 D.J. Flynn PS406 – Week 1 Section Spring 2014 2 / 24

Logistics Logistics Section: Wednesday, 5:00-6:00, Scott 212 Office Hours: Wednesday, 3:30-5:00, Scott 230 Grades: Labs (50%), Final Project (50%) Website: djflynn.org/teaching Rescheduling Reschedule section next week, 4/9 (I’m out of town) Meet somewhere else on 4/23 (no room available in Scott) We’ll do the L A T EX session afer the next section, 6:00-7:00 D.J. Flynn PS406 – Week 1 Section Spring 2014 3 / 24

Matrix algebra review Matrices A matrix is a rectangular array of numbers, symbols, or expressions, arranged in rows and columns. 1 The individual items in a matrix are called its elements or entries . 2 We describe matrices by how many rows and columns they have: rows by columns. This is a 4 x 2 matrix:   1 5       2 6            3 7           4 8  1 Thanks Wikipedia. 2 Elements/entries are usually numbers, but they can also be letters, images, etc. D.J. Flynn PS406 – Week 1 Section Spring 2014 4 / 24

Matrix algebra review Vectors and scalars One-dimensional matrices are called vectors . They come in two varieties: row vectors and column vectors :   2       4         � �   12 14 16 18 20  6            8            10  A matrix with one row and one column is a scalar . D.J. Flynn PS406 – Week 1 Section Spring 2014 5 / 24

Matrix algebra review Matrix operations We’ll talk briefly about: addition and subtraction 1 multiplication 2 transpose 3 inverse 4 Note: the homeworks won’t ask you to manipulate matrices. This is just so you can better understand the proofs in the slides. D.J. Flynn PS406 – Week 1 Section Spring 2014 6 / 24

Matrix algebra review Matrix addition/subtraction Two matrices must have same number of rows and columns Intuitive: given two matrices A and B , their sum, R , is simply: R = A + B For row i and column j , this is: r ij = a ij + b ij Example:  1 2   7 8   8 10                     +  = 3 4 9 10 12 14                                   5 6 11 12 16 18 D.J. Flynn PS406 – Week 1 Section Spring 2014 7 / 24

Matrix algebra review Matrix multiplication: scalar*matrix Intuitive: given scalar a and matrix B , the result, R is simply: R = aB For row i and column j , this is: r ij = ab ij Example: � � � � 2 6 16 48 8 = 3 7 24 56 D.J. Flynn PS406 – Week 1 Section Spring 2014 8 / 24

Matrix algebra review Matrix multiplication: row vector*column vector The row vector must have as many columns as the column vector has rows. The product of a row vector and column vector will always be a scalar. Given row vector a and column vector b , their product, r , is: n � r = a i b i i = 1 Example:  8      � �    = (2*8) + (6*1) + (3*4) = 34 2 6 3 1            4 D.J. Flynn PS406 – Week 1 Section Spring 2014 9 / 24

Matrix algebra review Matrix multiplication: matrix*matrix The first matrix must have as many columns as the second matrix has rows. (If matrices violate this rule, then multiplying them is illegal ). Multipling these matrices is legal:  − 3      � �   4 8 2            7  Multiplying these matrices is illegal:  8 5   2              6 1 8                      9 4   5  The resulting matrix will have as many rows as the first matrix and as many columns as the second matrix. D.J. Flynn PS406 – Week 1 Section Spring 2014 10 / 24

Matrix algebra review Matrix multiplication: matrix*matrix Given two matrices A and B , the i , j entry of AB is the inner product of the i th row of A and the j th column of B . Example:  b 11 b 12   a 11 a 12 a 13 a 14   ∗ ∗          b 21 b 22                 = a 21 a 22 a 23 a 24 ∗ ∗               b 31 b 32                 ( AB ) 32  a 31 a 32 a 33 a 34   ∗        b 41 b 42 ( AB ) 32 = a 31 b 12 + a 32 b 22 + a 33 b 32 + a 34 b 42 Let’s do an example with actual numbers on the board. D.J. Flynn PS406 – Week 1 Section Spring 2014 11 / 24

Matrix algebra review Matrix transpose The transpose of matrix A is denoted A T or A ′ . The first row of a matrix becomes the first column of the transpose matrix, the second row of the matrix becomes the second column of the transpose, etc. The transpose of a row vector will be a column vector, and the transpose of a column vector will be a row vector. Example:   1 2 � �   1 3 5   A T =   A= 3 4       2 4 6      5 6  D.J. Flynn PS406 – Week 1 Section Spring 2014 12 / 24

Matrix algebra review Matrix inverse Algebra review: the inverse of a number is that number which, when multiplied by the original number, gives a product of 1. The inverse of a matrix is that matrix which, when multiplied by the original matrix, gives the identity matrix:  1 0 0        I = 0 1 0             0 0 1 More formally, the matrix A is invertible if there exists a matrix A − 1 such that AA − 1 = I A − 1 A = I and Only square matrices (same number of rows and columns) have inverses – but not all square matrices have inverses. D.J. Flynn PS406 – Week 1 Section Spring 2014 13 / 24

Matrix algebra review Matrix inverse 3 There are several ways to find the inverse of a matrix. For large matrices, the process is time-consuming. That’s why we use R . For example, this is a shortcut for finding the inverse of a 2x2 matrix: D.J. Flynn PS406 – Week 1 Section Spring 2014 14 / 24

OLS review OLS Review y = X β + ǫ y is an N ∗ 1 vector of responses, X is an N ∗ p matrix of covariates, β is a p ∗ 1 vector of coefficients, ǫ is an N ∗ 1 vector of errors. D.J. Flynn PS406 – Week 1 Section Spring 2014 15 / 24

OLS review The OLS estimator in matrix form We know from lecture that ˆ β = ( X T X ) − 1 X T y � ( X i − ¯ X )( Y i − ¯ Y ) We know from last quarter that this equals � ( X i − ¯ X ) 2 D.J. Flynn PS406 – Week 1 Section Spring 2014 16 / 24

OLS review Suppose we have a dataset, N = 10 , p = 5 D.J. Flynn PS406 – Week 1 Section Spring 2014 17 / 24

OLS review The response vector: n x 1 D.J. Flynn PS406 – Week 1 Section Spring 2014 18 / 24

OLS review The response vector: n x 1   Y i 1        Y i 2            Y i 3           Y i 4             Y i 5           Y i 6             Y i 7           Y i 8            Y i 9            Y i 10   D.J. Flynn PS406 – Week 1 Section Spring 2014 18 / 24

OLS review The covariate matrix: n x p D.J. Flynn PS406 – Week 1 Section Spring 2014 19 / 24

OLS review The covariate matrix: n x p   1 X 1 , i 1 X 2 , i 1 X 3 , i 1 X 4 , i 1 X 5 , i 1        1 X 1 , i 2 X 2 , i 2 X 3 , i 2 X 4 , i 2 X 5 , i 2            X 1 , i 3 X 2 , i 3 X 3 , i 3 X 4 , i 3 X 5 , i 3 1           X 1 , i 4 X 2 , i 4 X 3 , i 4 X 4 , i 4 X 5 , i 4  1            X 1 , i 5 X 2 , i 5 X 3 , i 5 X 4 , i 5 X 5 , i 5 1           X 1 , i 6 X 2 , i 6 X 3 , i 6 X 4 , i 6 X 5 , i 6  1            1 X 1 , i 7 X 2 , i 7 X 3 , i 7 X 4 , i 7 X 5 , i 7           1 X 1 , i 8 X 2 , i 8 X 3 , i 8 X 4 , i 8 X 5 , i 8            1 X 1 , i 9 X 2 , i 9 X 3 , i 9 X 4 , i 9 X 5 , i 9            1 X 1 , i 10 X 2 , i 10 X 3 , i 10 X 4 , i 10 X 5 , i 10   D.J. Flynn PS406 – Week 1 Section Spring 2014 19 / 24

OLS review The parameter vector: k x 1 D.J. Flynn PS406 – Week 1 Section Spring 2014 20 / 24