CS70: Jean Walrand: Lecture 36. Gaussian and CLT CS70: Jean - PowerPoint PPT Presentation

Expectation of function of RV Definition The expectation of a function of a random variable is defined as

Expectation of function of RV Definition The expectation of a function of a random variable is defined as � ∞ E [ h ( X )] = − ∞ h ( x ) f X ( x ) dx .

Expectation of function of RV Definition The expectation of a function of a random variable is defined as � ∞ E [ h ( X )] = − ∞ h ( x ) f X ( x ) dx . Justification: Say X = n δ w.p. f X ( n δ ) δ .

Expectation of function of RV Definition The expectation of a function of a random variable is defined as � ∞ E [ h ( X )] = − ∞ h ( x ) f X ( x ) dx . Justification: Say X = n δ w.p. f X ( n δ ) δ . Then, E [ h ( X )] = ∑ h ( n δ ) Pr [ X = n δ ] n

Expectation of function of RV Definition The expectation of a function of a random variable is defined as � ∞ E [ h ( X )] = − ∞ h ( x ) f X ( x ) dx . Justification: Say X = n δ w.p. f X ( n δ ) δ . Then, E [ h ( X )] = ∑ h ( n δ ) Pr [ X = n δ ] = ∑ h ( n δ ) f X ( n δ ) δ n n

Expectation of function of RV Definition The expectation of a function of a random variable is defined as � ∞ E [ h ( X )] = − ∞ h ( x ) f X ( x ) dx . Justification: Say X = n δ w.p. f X ( n δ ) δ . Then, � ∞ E [ h ( X )] = ∑ h ( n δ ) Pr [ X = n δ ] = ∑ h ( n δ ) f X ( n δ ) δ = − ∞ h ( x ) f X ( x ) dx . n n � g ( x ) dx ≈ ∑ n g ( n δ ) δ with g ( x ) = h ( x ) f X ( x ) . Indeed,

Expectation of function of RV Definition The expectation of a function of a random variable is defined as � ∞ E [ h ( X )] = − ∞ h ( x ) f X ( x ) dx . Justification: Say X = n δ w.p. f X ( n δ ) δ . Then, � ∞ E [ h ( X )] = ∑ h ( n δ ) Pr [ X = n δ ] = ∑ h ( n δ ) f X ( n δ ) δ = − ∞ h ( x ) f X ( x ) dx . n n � g ( x ) dx ≈ ∑ n g ( n δ ) δ with g ( x ) = h ( x ) f X ( x ) . Indeed, Fact Expectation is linear.

Variance Definition: The variance of a continuous random variable X is defined as

Variance Definition: The variance of a continuous random variable X is defined as E (( X − E ( X )) 2 ) var [ X ] =

Variance Definition: The variance of a continuous random variable X is defined as E (( X − E ( X )) 2 ) = E ( X 2 ) − ( E ( X )) 2 var [ X ] =

Variance Definition: The variance of a continuous random variable X is defined as E (( X − E ( X )) 2 ) = E ( X 2 ) − ( E ( X )) 2 var [ X ] = � ∞ � � ∞ � 2 − ∞ x 2 f ( x ) dx − = − ∞ xf ( x ) dx .

Motivation for Gaussian Distribution Key fact:

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution.

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution. This is the Central Limit Theorem.

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution. This is the Central Limit Theorem. (See later.)

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution. This is the Central Limit Theorem. (See later.) Examples: Binomial and Poisson suitably scaled.

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution. This is the Central Limit Theorem. (See later.) Examples: Binomial and Poisson suitably scaled. This explains why the Gaussian distribution

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution. This is the Central Limit Theorem. (See later.) Examples: Binomial and Poisson suitably scaled. This explains why the Gaussian distribution (the bell curve)

Motivation for Gaussian Distribution Key fact: The sum of many small independent RVs has a Gaussian distribution. This is the Central Limit Theorem. (See later.) Examples: Binomial and Poisson suitably scaled. This explains why the Gaussian distribution (the bell curve) shows up everywhere.

Normal Distribution. For any µ and σ , a normal (aka Gaussian )

Normal Distribution. For any µ and σ , a normal (aka Gaussian ) random variable Y , which we write as Y = N ( µ , σ 2 ) , has pdf

Normal Distribution. For any µ and σ , a normal (aka Gaussian ) random variable Y , which we write as Y = N ( µ , σ 2 ) , has pdf 1 2 πσ 2 e − ( y − µ ) 2 / 2 σ 2 . f Y ( y ) = √

Normal Distribution. For any µ and σ , a normal (aka Gaussian ) random variable Y , which we write as Y = N ( µ , σ 2 ) , has pdf 1 2 πσ 2 e − ( y − µ ) 2 / 2 σ 2 . f Y ( y ) = √ Standard normal has µ = 0 and σ = 1 .

Normal Distribution. For any µ and σ , a normal (aka Gaussian ) random variable Y , which we write as Y = N ( µ , σ 2 ) , has pdf 1 2 πσ 2 e − ( y − µ ) 2 / 2 σ 2 . f Y ( y ) = √ Standard normal has µ = 0 and σ = 1 .

Normal Distribution. For any µ and σ , a normal (aka Gaussian ) random variable Y , which we write as Y = N ( µ , σ 2 ) , has pdf 1 2 πσ 2 e − ( y − µ ) 2 / 2 σ 2 . f Y ( y ) = √ Standard normal has µ = 0 and σ = 1 . Note: Pr [ | Y − µ | > 1 . 65 σ ] = 10 %;

Normal Distribution. For any µ and σ , a normal (aka Gaussian ) random variable Y , which we write as Y = N ( µ , σ 2 ) , has pdf 1 2 πσ 2 e − ( y − µ ) 2 / 2 σ 2 . f Y ( y ) = √ Standard normal has µ = 0 and σ = 1 . Note: Pr [ | Y − µ | > 1 . 65 σ ] = 10 %; Pr [ | Y − µ | > 2 σ ] = 5 % .

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) .

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . √

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] =

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] = Pr [ µ + σ X ∈ [ y , y + dy ]]

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] = Pr [ µ + σ X ∈ [ y , y + dy ]] = Pr [ σ X ∈ [ y − µ , y − µ + dy ]]

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] = Pr [ µ + σ X ∈ [ y , y + dy ]] = Pr [ σ X ∈ [ y − µ , y − µ + dy ]] Pr [ X ∈ [ y − µ , y − µ + dy = σ ]] σ σ

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] = Pr [ µ + σ X ∈ [ y , y + dy ]] = Pr [ σ X ∈ [ y − µ , y − µ + dy ]] Pr [ X ∈ [ y − µ , y − µ + dy = σ ]] σ σ f X ( y − µ ) dy = σ σ

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] = Pr [ µ + σ X ∈ [ y , y + dy ]] = Pr [ σ X ∈ [ y − µ , y − µ + dy ]] Pr [ X ∈ [ y − µ , y − µ + dy = σ ]] σ σ f X ( y − µ ) dy σ = 1 σ f X ( y − µ = ) dy σ σ

Scaling and Shifting Theorem Let X = N ( 0 , 1 ) and Y = µ + σ X . Then Y = N ( µ , σ 2 ) . 2 π exp {− x 2 1 Proof: f X ( x ) = 2 } . Now, √ f Y ( y ) dy = Pr [ Y ∈ [ y , y + dy ]] = Pr [ µ + σ X ∈ [ y , y + dy ]] = Pr [ σ X ∈ [ y − µ , y − µ + dy ]] Pr [ X ∈ [ y − µ , y − µ + dy = σ ]] σ σ f X ( y − µ ) dy σ = 1 σ f X ( y − µ = ) dy σ σ 2 πσ 2 exp {− ( y − µ ) 2 1 √ = } dy . 2 σ 2

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 .

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,....

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry.

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry. E [ X 2 ] var [ X ] =

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry. exp {− x 2 1 � E [ X 2 ] = x 2 var [ X ] = √ 2 } dx 2 π

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry. exp {− x 2 1 � E [ X 2 ] = x 2 var [ X ] = √ 2 } dx 2 π xd exp {− x 2 1 � √ = − 2 } 2 π

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry. exp {− x 2 1 � E [ X 2 ] = x 2 var [ X ] = √ 2 } dx 2 π xd exp {− x 2 exp {− x 2 1 1 � � √ √ = − 2 } = 2 } dx by IBP 2 π 2 π

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry. exp {− x 2 1 � E [ X 2 ] = x 2 var [ X ] = √ 2 } dx 2 π xd exp {− x 2 exp {− x 2 1 1 � � √ √ = − 2 } = 2 } dx by IBP 2 π 2 π � = f X ( x ) dx

Expectation, Variance. Theorem If Y = N ( µ , σ 2 ) , then E [ Y ] = µ and var [ Y ] = σ 2 . Proof: It suffices to show the result for X = N ( 0 , 1 ) since Y = µ + σ X ,.... 2 π exp {− x 2 1 Thus, f X ( x ) = 2 } . √ First note that E [ X ] = 0 , by symmetry. exp {− x 2 1 � E [ X 2 ] = x 2 var [ X ] = √ 2 } dx 2 π xd exp {− x 2 exp {− x 2 1 1 � � √ √ = − 2 } = 2 } dx by IBP 2 π 2 π � = f X ( x ) dx = 1 .

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.”

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 .

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n .

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n .

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . E ( A ′ n )

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . 1 E ( A ′ n ) = σ / √ n ( E ( A n ) − µ )

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . 1 E ( A ′ n ) = σ / √ n ( E ( A n ) − µ ) = 0 .

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . 1 E ( A ′ n ) = σ / √ n ( E ( A n ) − µ ) = 0 . Var ( A ′ n )

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . 1 E ( A ′ n ) = σ / √ n ( E ( A n ) − µ ) = 0 . Var ( A ′ 1 n ) = σ 2 / n Var ( A n )

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . 1 E ( A ′ n ) = σ / √ n ( E ( A n ) − µ ) = 0 . Var ( A ′ 1 n ) = σ 2 / n Var ( A n ) = 1 .

Central limit theorem. Law of Large Numbers: For any set of independent identically distributed random variables, X i , A n = 1 n ∑ X i “tends to the mean.” Say X i have expecation µ = E ( X i ) and variance σ 2 . Mean of A n is µ , and variance is σ 2 n . n = A n − µ Let A ′ σ / √ n . 1 E ( A ′ n ) = σ / √ n ( E ( A n ) − µ ) = 0 . Var ( A ′ 1 n ) = σ 2 / n Var ( A n ) = 1 . Central limit theorem: As n goes to infinity the distribution of A ′ n approaches the standard normal distribution. � α 1 ∞ e − x 2 / 2 dx . Pr [ A ′ √ n ≤ α ] → 2 π

Coins and normal.. Let X 1 , X 2 ,... be i.i.d. B ( p ) .