MLES & Multivariate Normal Theory STA721 Linear Models Duke - PowerPoint PPT Presentation

Geometric View Multivariate Normal MLES & Multivariate Normal Theory STA721 Linear Models Duke University Merlise Clyde September 1, 2015 duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Geometric View Fitted Values ˆ Y = P X Y = X ˆ β Residuals e = ( I − P X ) Y Y = ˆ Y + e � Y � 2 = � P X Y � 2 + � ( I − P X ) Y � 2 duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Properties ˆ Y = ˆ µ is an unbiased estimate of µ = X β E[ˆ Y ] = E[P X Y ] = P X E[ Y ] = P X µ = µ E[ e ] = 0 if µ ∈ C ( X ) E[ e ] = E[( I − P X ) Y ] = ( I − P X )E[ Y ] = ( I − P X ) µ = 0 Will not be 0 if µ / ∈ C ( X ) (useful for model checking) duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Estimate of σ 2 MLE of σ 2 : σ 2 = e T e = Y T ( I − P X ) Y ˆ n n Is this an unbiased estimate of σ 2 ? Need expectations of quadratic forms Y T AY for A an n × n matrix Y a random vector in R n duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Quadratic Forms Without loss of generality we can assume that A = A T Y T AY is a scalar Y T AY = ( Y T AY ) T = Y T A T Y Y T AY + Y T A T Y Y T AY = 2 Y T ( A + A T ) Y T AY Y = 2 may take A = A T duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Expectations of Quadratic Forms Theorem Let Y be a random vector in R n with E [ Y ] = µ and Cov ( Y ) = Σ . Then E [ Y T AY ] = tr A Σ + µ T A µ . Result useful for finding expected values of Mean Squares; no normality required! duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Proof Start with ( Y − µ ) T A ( Y − µ ), expand and take expectations E[( Y − µ ) T A ( Y − µ )] E[ Y T AY + µ T A µ − µ T AY − Y T A µ ] = E[ Y T AY ] + µ T A µ − µ T A µ − µ T A µ = E[ Y T AY ] − µ T A µ = Rearrange E[ Y T AY ] E[( Y − µ ) T A ( Y − µ )] + µ T A µ = E[tr( Y − µ ) T A ( Y − µ )] + µ T A µ = E[tr A ( Y − µ )( Y − µ ) T ] + µ T A µ = trE[ A ( Y − µ )( Y − µ ) T ] + µ T A µ = tr A E([( Y − µ )( Y − µ ) T ] + µ T A µ = tr AΣ + µ T A µ = tr A ≡ � n i =1 a ii duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal σ 2 Expectation of ˆ Use the theorem: E[ Y T ( I − P X ) Y ] tr( I − P X ) σ 2 I + µ T ( I − P X ) µ = σ 2 tr( I − P X ) = σ 2 r ( I − P X ) = σ 2 ( n − r ( X )) = Therefore an unbiased estimate of σ 2 is e T e n − r ( X ) If X is full rank ( r ( X ) = p ) and P X = X ( X T X ) − 1 X T then the tr( X ( X T X ) − 1 X T ) tr(P X ) = tr( X T X ( X T X ) − 1 ) = duke.eps = tr( I p ) = p STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Spectral Theorem Theorem If A (n × n) is a symmetric real matrix then there exists a U (n × n) such that U T U = UU T = I n and a diagonal matrix Λ with elements λ i such that A = UΛU T U is an orthogonal matrix; U − 1 = U T The columns of U from an Orthonormal Basis for R n rank of A equals the number of non-zero eigenvalues λ i Columns of U associated with non-zero eigenvalues form an ONB for C ( A ) (eigenvectors of A ) A p = UΛ p U T (matrix powers) a square root of A > 0 is UΛ 1 / 2 U T duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Projections Projection Matrix If P is an orthogonal projection matrix, then its eigenvalues λ i are either zero or one with tr(P) = � i ( λ i ) = r (P) P = UΛU T P = P 2 ⇒ UΛU T UΛU T = UΛ 2 U T Λ = Λ 2 is true only for λ i = 1 or λ i = 0 Since r (P) is the number of non-zero eigenvalues, r (P) = � λ i = tr(P) � I r � � U T � 0 = U P U T P = [ U P U P ⊥ ] P U T P 0 0 n − r P ⊥ r � u i u T P = i i =1 sum of r rank 1 projections. duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Distributions Distribution of ˆ β Distribution of P X Y Distribution of e Distribution ot ˆ σ 2 duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Univariate Normal Definition We say that Z has a standard Normal distribution Z ∼ N(0 , 1) with mean 0 and variance 1 if it has density 1 e − 1 2 z 2 √ f Z ( z ) = 2 π If Y = µ + σ Z then Y ∼ N ( µ, σ 2 ) with mean µ and variance σ 2 1 2 2 πσ 2 e − 1 2 ( z − µ σ ) f Y ( y ) = √ duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Standard Multivariate Normal iid Let z i ∼ N(0 , 1) for i = 1 , . . . , d and define   z 1 z 2   Z ≡ .   .   .   z d Density of Z : 2 π e − z 2 = � d 1 i / 2 f Z ( z ) √ j =1 2 ( Z T Z ) = (2 π ) − d / 2 e − 1 E[ Z ] = 0 and Cov[ Z ] = I d Z ∼ N( 0 d , I d ) duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Multivariate Normal For a d dimensional multivariate normal random vector, we write Y ∼ N d ( µ , Σ ) E[ Y ] = µ : d dimensional vector with means E [ Y j ] Cov[ Y ] = Σ : d × d matrix with diagonal elements that are the variances of Y j and off diagonal elements that are the covariances E[( Y j − µ j )( Y k − µ k )] Density If Σ is positive definite ( x ′ Σx > 0 for any x � = 0 in R d ) then Y has a density a p ( Y ) = (2 π ) − d / 2 | Σ | − 1 / 2 exp( − 1 2( Y − µ ) T Σ − 1 ( Y − µ )) a with respect to Lebesgue measure on R d duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Multivariate Normal Density Density of Z ∼ N( 0 , I d ): 2 π e − z 2 = � d 1 i / 2 f Z ( z ) √ j =1 2 ( Z T Z ) = (2 π ) − d / 2 e − 1 Write Y = µ + AZ Solve for Z = g ( Y ) Jacobian of the transformation J ( Z → Y ) = | ∂ g ∂ Y | substitute g ( Y ) for Z in density and multiply by Jacobian f Y ( y ) = f Z ( z ) J ( Z → Y ) duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Multivariate Normal Density Y = µ + AZ for Z ∼ N( 0 , I d ) (1) Proof. since Σ > 0 , ∃ an A ( d × d ) such that A > 0 and AA T = Σ A > 0 ⇒ A − 1 exists Multiply both sides (1) by A − 1 : A − 1 Y = A − 1 µ + A − 1 AZ Rearrange A − 1 ( Y − µ ) = Z Jacobian of transformation d Z = | A − 1 | d Y Substitute and simplify algebra f ( Y ) = (2 π ) − d / 2 | Σ | − 1 / 2 exp( − 1 2( Y − µ ) T Σ − 1 ( Y − µ )) duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Singular Case Y = µ + AZ with Z ∈ R d and A is n × d E[ Y ] = µ Cov( Y ) = AA T ≥ 0 Y ∼ N( µ , Σ ) where Σ = AA T If Σ is singular then there is no density (on R n ), but claim that Y still has a multivariate normal distribution! Definition Y ∈ R n has a multivariate normal distribution N( µ , Σ ) if for any v ∈ R n v T Y has a normal distribution with mean v T µ and variance v T Σv see Lessons in Sakai for videos using Characteristic functions duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Linear Transformations are Normal If Y ∼ N n ( µ , Σ ) then for A m × n AY ∼ N m ( A µ , AΣA T ) AΣA T does not have to be positive definite! duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Equal in Distribution Multiple ways to define the same normal: Z 1 ∼ N( 0 , I n ), Z 1 ∈ R n and take A d × n Z 2 ∼ N( 0 , I p ), Z 2 ∈ R p and take B d × p Define Y = µ + AZ 1 Define W = µ + BZ 2 Theorem If Y = µ + AZ 1 and W = µ + BZ 2 then Y D = W if and only if AA T = BB T = Σ duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Zero Correlation and Independence Theorem For a random vector Y ∼ N ( µ , Σ ) partitioned as � Y 1 �� µ 1 � Σ 11 � � �� Σ 12 Y = ∼ N , Y 2 Σ 21 Σ 22 µ 2 then Cov ( Y 1 , Y 2 ) = Σ 12 = Σ T 21 = 0 if and only if Y 1 and Y 2 are independent. duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory

Geometric View Multivariate Normal Independence Implies Zero Covariance Proof. Cov( Y 1 , Y 2 ) = E[( Y 1 − µ 1 )( Y 2 − µ 2 ) T ] If Y 1 and Y 2 are independent E[( Y 1 − µ 1 )( Y 2 − µ 2 ) T ] = E[( Y 1 − µ 1 )E( Y 2 − µ 2 ) T ] = 00 T = 0 therefore Σ 12 = 0 duke.eps STA721 Linear Models Duke University MLES & Multivariate Normal Theory