Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced - PowerPoint PPT Presentation

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4 - 7/21/2011 Slide 1 of 41

Last Time ■ Matrices and vectors ◆ Eigenvalues Overview ● Last Time ◆ Eigenvectors ● Today’s Lecture MVN ◆ Determinants MVN Properties ■ Basic descriptive statistics using matrices: MVN Parameters ◆ Mean vectors MVN Likelihood Functions MVN in Common Methods ◆ Covariance Matrices Assessing Normality ◆ Correlation Matrices Wrapping Up Lecture #4 - 7/21/2011 Slide 2 of 41

Today’s Lecture ■ Putting our new knowledge to use with a useful statistical distribution: the Multivariate Normal Distribution Overview ■ This roughly maps onto Chapter 4 of Johnson and Wichern ● Last Time ● Today’s Lecture MVN MVN Properties MVN Parameters MVN Likelihood Functions MVN in Common Methods Assessing Normality Wrapping Up Lecture #4 - 7/21/2011 Slide 3 of 41

Multivariate Normal Distribution ■ The generalization of the univariate normal distribution to multiple variables is called the multivariate normal distribution (MVN) Overview MVN ■ Many multivariate techniques rely on this distribution in some ● Univariate Review ● MVN manner ● MVN Contours MVN Properties ■ Although real data may never come from a true MVN, the MVN Parameters MVN provides a robust approximation, and has many nice MVN Likelihood Functions mathematical properties MVN in Common Methods Assessing Normality ■ Furthermore, because of the central limit theorem, many Wrapping Up multivariate statistics converge to the MVN distribution as the sample size increases Lecture #4 - 7/21/2011 Slide 4 of 41

Univariate Normal Distribution ■ The univariate normal distribution function is: 1 2 πσ 2 e − [( x − µ ) /σ ] 2 / 2 f ( x ) = √ Overview MVN ● Univariate Review ● MVN ● MVN Contours ■ The mean is µ MVN Properties MVN Parameters ■ The variance is σ 2 MVN Likelihood Functions MVN in Common Methods ■ The standard deviation is σ Assessing Normality Wrapping Up ■ Standard notation for normal distributions is N ( µ, σ 2 ) , which will be extended for the MVN distribution Lecture #4 - 7/21/2011 Slide 5 of 41

Univariate Normal Distribution N (0 , 1) Univariate Normal Distribution Overview MVN 0.4 ● Univariate Review ● MVN ● MVN Contours MVN Properties 0.3 MVN Parameters MVN Likelihood Functions MVN in Common Methods 0.2 f(x) Assessing Normality Wrapping Up 0.1 0.0 −6 −4 −2 0 2 4 6 x Lecture #4 - 7/21/2011 Slide 6 of 41

Univariate Normal Distribution N (0 , 2) Univariate Normal Distribution Overview MVN 0.4 ● Univariate Review ● MVN ● MVN Contours MVN Properties 0.3 MVN Parameters MVN Likelihood Functions MVN in Common Methods f(x) 0.2 Assessing Normality Wrapping Up 0.1 0.0 −6 −4 −2 0 2 4 6 x Lecture #4 - 7/21/2011 Slide 7 of 41

Univariate Normal Distribution N (1 . 75 , 1) Univariate Normal Distribution Overview MVN 0.4 ● Univariate Review ● MVN ● MVN Contours MVN Properties 0.3 MVN Parameters MVN Likelihood Functions MVN in Common Methods 0.2 f(x) Assessing Normality Wrapping Up 0.1 0.0 −6 −4 −2 0 2 4 6 x Lecture #4 - 7/21/2011 Slide 8 of 41

UVN - Notes ■ The area under the curve for the univariate normal distribution is a function of the variance/standard deviation Overview ■ In particular: MVN ● Univariate Review ● MVN P ( µ − σ ≤ X ≤ µ + σ ) = 0 . 683 ● MVN Contours MVN Properties P ( µ − 2 σ ≤ X ≤ µ + 2 σ ) = 0 . 954 MVN Parameters MVN Likelihood Functions MVN in Common Methods ■ Also note the term in the exponent: Assessing Normality � 2 Wrapping Up � ( x − µ ) = ( x − µ )( σ 2 ) − 1 ( x − µ ) σ ■ This is the square of the distance from x to µ in standard deviation units, and will be generalized for the MVN Lecture #4 - 7/21/2011 Slide 9 of 41

MVN ■ The multivariate normal distribution function is: 1 − 1 ( x − µ ) / 2 (2 π ) p/ 2 | Σ | 1 / 2 e − ( x − µ ) ′ Σ f ( x ) = Overview MVN ● Univariate Review ● MVN ● MVN Contours ■ The mean vector is µ MVN Properties MVN Parameters ■ The covariance matrix is Σ MVN Likelihood Functions MVN in Common Methods ■ Standard notation for multivariate normal distributions is Assessing Normality N p ( µ , Σ ) Wrapping Up ■ Visualizing the MVN is difficult for more than two dimensions, so I will demonstrate some plots with two variables - the bivariate normal distribution Lecture #4 - 7/21/2011 Slide 10 of 41

Bivariate Normal Plot #1 � � � � 0 1 0 µ = , Σ = 0 0 1 Overview MVN ● Univariate Review ● MVN ● MVN Contours 0.16 0.14 MVN Properties 0.12 MVN Parameters 0.1 MVN Likelihood Functions 0.08 0.06 MVN in Common Methods 0.04 Assessing Normality 0.02 Wrapping Up 0 4 2 4 2 0 0 −2 −2 −4 −4 Lecture #4 - 7/21/2011 Slide 11 of 41

Bivariate Normal Plot #1a � � � � 0 1 0 µ = , Σ = 0 0 1 Overview 4 MVN ● Univariate Review ● MVN 3 ● MVN Contours MVN Properties 2 MVN Parameters 1 MVN Likelihood Functions 0 MVN in Common Methods Assessing Normality −1 Wrapping Up −2 −3 −4 −4 −3 −2 −1 0 1 2 3 4 Lecture #4 - 7/21/2011 Slide 12 of 41

Bivariate Normal Plot #2 � � � � 0 1 0 . 5 µ = , Σ = 0 0 . 5 1 Overview MVN ● Univariate Review ● MVN ● MVN Contours MVN Properties 0.2 MVN Parameters 0.15 MVN Likelihood Functions MVN in Common Methods 0.1 Assessing Normality 0.05 Wrapping Up 0 4 2 4 2 0 0 −2 −2 −4 −4 Lecture #4 - 7/21/2011 Slide 13 of 41

Bivariate Normal Plot #2 � � � � 0 1 0 . 5 µ = , Σ = 0 0 . 5 1 Overview 4 MVN ● Univariate Review ● MVN 3 ● MVN Contours MVN Properties 2 MVN Parameters 1 MVN Likelihood Functions 0 MVN in Common Methods Assessing Normality −1 Wrapping Up −2 −3 −4 −4 −3 −2 −1 0 1 2 3 4 Lecture #4 - 7/21/2011 Slide 14 of 41

MVN Contours ■ The lines of the contour plots denote places of equal probability mass for the MVN distribution ◆ The lines represent points of both variables that lead to Overview the same height on the z-axis (the height of the surface) MVN ● Univariate Review ● MVN ● MVN Contours ■ These contours can be constructed from the eigenvalues MVN Properties and eigenvectors of the covariance matrix MVN Parameters ◆ The direction of the ellipse axes are in the direction of the MVN Likelihood Functions eigenvalues MVN in Common Methods Assessing Normality ◆ The length of the ellipse axes are proportional to the Wrapping Up constant times the eigenvector ■ Specifically: ( x − µ ) ′ Σ − 1 ( x − µ ) = c 2 has ellipsoids centered at µ , and has axes ± c √ λ i e i Lecture #4 - 7/21/2011 Slide 15 of 41

MVN Contours, Continued ■ Contours are useful because they provide confidence regions for data points from the MVN distribution Overview ■ The multivariate analog of a confidence interval is given by MVN an ellipsoid, where c is from the Chi-Squared distribution ● Univariate Review ● MVN with p degrees of freedom ● MVN Contours MVN Properties ■ Specifically: MVN Parameters MVN Likelihood Functions ( x − µ ) ′ Σ − 1 ( x − µ ) = χ 2 p ( α ) MVN in Common Methods Assessing Normality Wrapping Up provides the confidence region containing 1 − α of the probability mass of the MVN distribution Lecture #4 - 7/21/2011 Slide 16 of 41

MVN Contour Example ■ Imagine we had a bivariate normal distribution with: � � � � 0 1 0 . 5 µ = , Σ = Overview 0 0 . 5 1 MVN ● Univariate Review ● MVN ■ The covariance matrix has eigenvalues and eigenvectors: ● MVN Contours MVN Properties � � � � 1 . 5 0 . 707 − 0 . 707 MVN Parameters λ = , E = 0 . 5 0 . 707 0 . 707 MVN Likelihood Functions MVN in Common Methods ■ We want to find a contour where 95% of the probability will Assessing Normality fall, corresponding to χ 2 2 (0 . 05) = 5 . 99 Wrapping Up Lecture #4 - 7/21/2011 Slide 17 of 41

MVN Contour Example ■ This contour will be centered at µ ■ Axis 1: Overview MVN � � � � � � √ 0 . 707 2 . 12 − 2 . 12 ● Univariate Review µ ± 5 . 99 × 1 . 5 = , ● MVN 0 . 707 2 . 12 − 2 . 12 ● MVN Contours MVN Properties MVN Parameters ■ Axis 2: MVN Likelihood Functions MVN in Common Methods � � � � � � Assessing Normality √ − 0 . 707 − 1 . 22 1 . 22 µ ± 5 . 99 × 0 . 5 = , Wrapping Up 0 . 707 1 . 22 − 1 . 22 Lecture #4 - 7/21/2011 Slide 18 of 41

MVN Properties ■ The MVN distribution has some convenient properties ■ If X has a multivariate normal distribution, then: Overview 1. Linear combinations of X are normally distributed MVN MVN Properties ● MVN Properties 2. All subsets of the components of X have a MVN MVN Parameters distribution MVN Likelihood Functions 3. Zero covariance implies that the corresponding MVN in Common Methods components are independently distributed Assessing Normality Wrapping Up 4. The conditional distributions of the components are MVN Lecture #4 - 7/21/2011 Slide 19 of 41