Foundations of Computing II Lecture 22: Moments Stefano Tessaro - PowerPoint PPT Presentation

CSE 312 Foundations of Computing II Lecture 22: Moments Stefano Tessaro tessaro@cs.washington.edu 1

Things we mentioned, but did not prove: • If ! ∼ #(%, ' ( ) , then *! + , ∼ #(*% + ,, * ( ' ( ) . ( , then ! - + ! ( ∼ ( ) and ! ( ∼ # % ( , ' ( • If ! - ∼ #(% - , ' - ( + ' ( ( ) . #(% - + % ( , ' - • The Central Limit Theorem (CLT). (Aka. “Everything” converges to a Gaussian!) 2

Reminder 5 6 2 . / = 1 7! 234 We are going to use this many times today! 3

Moments Definition. The 9 -th moment of a random variable ! is : ! ; . 1st moment = expectation : ! 1st moment and 2nd moment → variance Var ! = : ! ( − : ! ( Generally, a random variable is determined uniquely by its moments. … let’s make this more formal! 4

Moment Generating Functions Definition. The moment generating function (MGF) of ! is the function @ A : ℝ → ℝ @ A E = : . FA . 5 E! 2 @ A E = : . FA = : 1 7! 234 = 1 + : ! E + : ! ( E ( 2 + : ! I E I 6 + ⋯ 5

MGFs – Basic Properties Theorem. ! and L are identically distributed if and only if @ A = @ M . Theorem. If ! and L are independent, then for all E ∈ ℝ , @ AOM E = @ A E ⋅ @ M (E) Proof. @ AOM E = : . F(AOM) = : . FA . FM = : . FA ⋅ : . FM = @ A E ⋅ @ M (E) 6

Example – MGF of Poisson Recall: ! ∼ Poi T : ℙ ! = 7 = . WX X Y 2! 5 @ A E = : . FA = 1 ℙ ! = 7 ⋅ . F2 234 5 (. F T) 2 5 . WX T 2 7! ⋅ . F2 = . WX 1 = 1 7! 234 234 = . WX . XZ [ = . X(Z [ W-) 7

Example – Sum of Poissons Claim. If ! - ∼ Poi(T - ) and ! ( ∼ Poi T ( are independent, then ! - + ! ( ∼ Poi(T - + T ( ) Proof. @ A \ E = . X \ (Z [ W-) Previous Reminder: If ! and L are independent, then @ AOM E = @ A E ⋅ @ M (E) slide @ A ] E = . X ] (Z [ W-) @ A \ OA ] E = @ A \ E ⋅ @ A ] E = . X \ (Z [ W-) ⋅ . X ] Z [ W- = e (X \ OX ] )(Z [ W-) ! - + ! ( ∼ Poi(T - + T ( ) 8

MGF of the Normal Distribution Theorem. If !~#(%, ' ( ) , then @ A E = . F`O []a] ] We will prove it below, but first, some interesting consequences! 9

Theorem. If !~#(%, ' ( ) , then @ A E = . F`O []a] ] MGF of Normal – Applications Fact 1. If !~#(%, ' ( ) , then L = *! + , ∼ #(*% + ,, * ( ' ( ) @ A E = . F`OF ] b ] ( = : . FcA . Fd @ M E = : . F cAOd = . Fd :(. Fc A ) = . Fd @ A (E*) = . F(c`Od)OF ] c ] b ] MGF of #(*% + = . Fd . Fc`OF ] c ] b ] ( ,, * ( ' ( ) ( 10

Theorem. If !~#(%, ' ( ) , then @ A E = . F`O []a] ] MGF of Normal – Applications ( ) , then Fact 2. If ! - , … , ! f independent and ! 2 ∼ #(% 2 , ' 2 ( + ⋯ + ' f ( ) ! - + ⋯ + ! f ∼ #(% - + ⋯ + % f , ' - @ A \ O⋯OA g E = @ A \ E ⋯ @ A g (E) = . F(` \ O ⋯O` g )OF ] (b \ ] O⋯Ob g ] ) ] = . F` \ OF ] b \ ⋯ . F` g OF ] b g ] ( ( ( ( + ⋯ + ' f ( ) MGF of #(% - + ⋯ + % f , ' - Try a direct proof for both facts? 11

MGF of the Normal Distribution – Proof – Standard Normal - (j . W/ ] /( Recall: If !~#(0,1) , then i A (6) = O5 O5 1 1 @ A E = : . FA = . F/ . W/ ] /( d6 . F/W/ ] /( d6 m = m 2l 2l W5 W5 = E ( − 6 − E ( E6 − 6 ( = 2E6 − 6 ( 2 2 2 O5 O5 F ] 1 1 ( W /WF ] /( d6 = . F ] /( . W /WF ] /( d6 = . F ] /( = m . m 2l 2l W5 W5 12

MGF of the Normal Distribution – General Proof (1/2) - (jb . W /W` ] /(b ] Recall: If !~#(%, ' ( ) , then i A (6) = O5 1 @ A E = : . FA = . F/ . W /W` ] /(b ] d6 m 2l' W5 O5 1 . F(obO`)Wo ] /( d6 = m 2l' W5 O5 1 p = 6 − % . FobWo ] /( d6 = . F` m 2l' ' W5 O5 1 . FobWo ] /( d6 = . F` m dp dp 2l' W5 13

MGF of the Normal Distribution – General Proof (2/2) - (jb . W /W` ] /(b ] Recall: If !~#(%, ' ( ) , then i A (6) = O5 1 . FobWo ] /( d6 d6 p = 6 − % dp = ' = . F` @ A E = : . FA m dp dp ' 2l' W5 O5 1 Rewrite x as a . FobWo ] /( 'dp = . F` m function of z, 2l' W5 and take O5 1 derivative! . FobWo ] /( dp = . F` m 2l W5 F ] b ] = . F`OF ] b ] = . F` . ( ( 14

Proof of the CLT f = ! - + ⋯ + ! f − v% L Theorem. (Central Limit Theorem) The CDF of L f converges to ' v the CDF of the standard normal #(0,1) , i.e., u 1 ! - , … , ! f iid with mean % . W/ ] /( d6 f→5 ℙ L lim f ≤ t = m 2l and variance ' ( W5 g E → . F ] /( as v → ∞ Proof shows that @ M Let’s do this for the case ' = 1 and % = 0 . f = ! - + ⋯ + ! f g E = :(. FA ) = :(. F(A \ O⋯OA g )/ f ) @ M L v f ! has same f :(. FA Y / f ) = :(. FA/ f ) = x distribution as ! - , … , ! f 23- 15

f g E = :(. FA/ f ) @ M Proof of the CLT v + : ! ( E ( E I E :(. FA/ f ) = 1 + : ! 2v + : ! I 6v -.z + ⋯ 1 = ' ( = : ! ( − : ! ( = :(! ( ) But note that: : ! = 0 :(. FA/ f ) = 1 + E ( E I E ( 2v + : ! I 6v -.z + : ! { 24v ( + ⋯ 2v 1 + : ! I E 3v 4.z + : ! { E ( = 1 + E ( ≈ 1 + E ( as v → ∞ + ⋯ 12v 2v f 1 + E ( f → . F ] /( as v → ∞ g E = :(. FA/ f ) ≈ @ M 2v 16