Cochrans Theorem . Yang Feng . . . . . . . . . . . . . - PowerPoint PPT Presentation

. Cochran’s Theorem . Yang Feng . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 1 / 22

Importance of Cochran’s Theorem Cochran’s theorem tells us about the distributions of partitioned sums of squares of normally distributed random variables. Traditional linear regression analysis relies upon making statistical claims about the distribution of sums of squares of normally distributed random variables (and ratios between them) In the simple normal regression model: ∑ ( Y i − ˆ Y i ) 2 SSE ∼ χ 2 ( n − 2) = σ 2 σ 2 Where does this come from? . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 2 / 22

Outline Establish the fact that the multivariate Gaussian sum of squares is χ 2 ( n ) distributed Provide intuition for Cochran’s theorem Prove a lemma in support of Cochran’s theorem Prove Cochran’s theorem Connect Cochran’s theorem back to matrix linear regression . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 3 / 22

χ 2 distribution Theorem 1: Suppose Z i are i . i . d . N (0 , 1), we have n Z 2 i ∼ χ 2 ( n ) ∑ i =1 . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 4 / 22

Proof: Z 2 i ∼ χ 2 (1) If Y 1 , · · · , Y n are i.i.d. random variables with moment generating functions (MGF) m Y 1 ( t ) , · · · , m Y n ( t ). Then the moment generating function for U = Y 1 + · · · + Y n is m U ( t ) = m Y 1 ( t ) × m Y 2 ( t ) · · · × m Y n ( t ) MGF fully characterize the distribution The MGF for χ 2 ( n ) is (1 − 2 t ) n / 2 . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 5 / 22

Quadratic Forms and Cochran’s Theorem Quadratic forms of normal random variables are of great importance in many branches of statistics Least Squares ANOVA Regression Analysis General idea: Split the sum of the squares of observations into a number of quadratic forms where each corresponds to some cause of variation . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 6 / 22

Quadratic Forms and Cochrans Theorem The conclusion of Cochran’s theorem is that, under the assumption of normality, the various quadratic forms are independent and χ 2 distributed. This fact is the foundation upon which many statistical tests rest. . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 7 / 22

Preliminaries: A Common Quadratic Form Let X ∼ N ( µ , Λ ) Consider the quadratic form that appears in the exponent of the normal density ( X − µ ) ′ Λ − 1 ( X − µ ) In the special case of µ = 0 and Λ = I , this reduces to X ′ X which by what we just proved we know is χ 2 ( n ) distributed Let’s prove it holds in the general case . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 8 / 22

Lemma 1 Let X ∼ N ( µ , Λ ) with | Λ | > 0 and n is the dimension of X , then ( X − µ ) ′ Λ − 1 ( X − µ ) ∼ χ 2 ( n ) . Proof . Let Y = Λ − 1 / 2 ( X − µ ), then we have Y ∼ N ( 0 , I ). Then, ( X − µ ) ′ Λ − 1 ( X − µ ) = Y ′ Y ∼ χ 2 ( n ) . . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 9 / 22

Cochran’s Theorem Let X 1 , X 2 , · · · , X n be i.i.d. N (0 , σ 2 )- distributed random variables, and suppose that n X 2 ∑ i = Q 1 + Q 2 + · · · + Q k , i =1 where Q 1 , Q 2 , · · · , Q k are positive semi-definite quadratic forms in X 1 , X 2 , · · · , X n , i.e., Q i = X ′ A i X , i = 1 , 2 , · · · , k Set r i = rank( A i ). If r 1 + r 2 + · · · + r k = n , then . . 1 Q 1 , Q 2 , · · · , Q k are independent. . . 2 Q i ∼ σ 2 χ 2 ( r i ) . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 10 / 22

Several linear algebra results X be a normal random vector. The components of X are independent if and only if they are uncorrelated. Let X ∼ N ( µ , Λ ), then Y = C ′ X ∼ N ( C ′ µ , C ′ ΛC ). We can find an orthogonal matrix C such that D = C ′ ΛC is a diagonal matrix. (Eigen Value Decomposition for Semi Positive Definite Matrix) The components of Y will be independent and var( Y k ) = λ k , where λ 1 , · · · , λ n are the eigenvalues of Λ . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 11 / 22

Lemma 2 Let X 1 , X 2 , · · · , X n be real numbers. Suppose that ∑ X 2 i can be split into a sum of positive semi-definite quadratic forms, that is ∑ X 2 i = Q 1 + Q 2 + · · · + Q k where Q i = X ′ A i X with rank( A i ) = r i . If ∑ r i = n , then there exists an orthogonal matrix C such that, with X = CY , we have Y 2 1 + Y 2 2 + · · · + Y 2 = Q 1 r 1 Y 2 r 1 +1 + Y 2 r 1 +2 + · · · + Y 2 Q 2 = r 1 + r 2 . . . Y 2 n − r k +1 + Y 2 n − r k +2 + · · · + Y 2 = Q k n . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 12 / 22

Remark Different quadratic forms contain different Y -variables and that the number of terms in each Q i equals that rank, r i , of Q i The Y 2 i end up in different sums, we’ll use this to prove independence of the different quadratic forms. Just prove for n = 2 case, the general case can be obtained by induction. . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 13 / 22

Proof For n = 2, we have Q = X ′ A 1 X + X ′ A 2 X There exists an orthogonal matrix C such that C ′ A 1 C = D , where D is a diagonal matrix with eigenvalues of A 1 . Since rank ( A 1 ) = r 1 , r 1 eigenvalues are positive and n − r 1 eigenvalues are 0. Suppose without loss of generality, the first r 1 eigenvalues are positive. Set X = CY , then we have X ′ X = Y ′ C ′ CY = Y ′ Y . . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 14 / 22

Proof Therefore, Q = ∑ n i =1 Y 2 i = ∑ r 1 i =1 λ i Y 2 i + Y ′ C ′ A 2 CY Then, rearranging the terms we have r 1 n ∑ (1 − λ i ) Y 2 ∑ Y 2 i + i = Y ′ C ′ A 2 CY i =1 i = r 1 +1 Since rank ( A 2 ) = r 2 = n − r 1 , we conclude that λ 1 = λ 2 = · · · = λ r 1 = 1 r 1 n Y 2 Y 2 ∑ ∑ Q 1 = i , Q 2 = i i =1 i = r 1 +1 . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 15 / 22

From this Lemma This lemma is about real numbers, not random variables It says that ∑ X 2 i can be split into a sum of positive semi-definite quadratic forms, then there is a orthogonal transformation X = CY such that each of the quadratic forms have nice properties: Each Y i appears in only one resulting sum of squares, which leads to the independence of the sum of squares. . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Yang Feng (Columbia University) Cochran’s Theorem 16 / 22