likelihood and point estimation lecture 09 biostatistics
play

Likelihood and Point Estimation Lecture 09 Biostatistics 602 - - PowerPoint PPT Presentation

Oct 23, 2022 •338 likes •1.44k views

. Summary February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang February 7th, 2013 Hyun Min Kang Likelihood and Point Estimation Lecture 09 Biostatistics 602 - Statistical Inference . . . . MLE Method of Moments Likelihood

  1. . MLE February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang i.i.d. Point Estimation : Ingredients . . Summary 6 / 24 Method of Moments . . . . Likelihood Function . . . . . . . . . . . . . . . . . . . . . . • Data: x = ( x 1 , · · · , x n ) - realizations of random variables ( X 1 , · · · , X n ) . • X 1 , · · · , X n ∼ f X ( x | θ ) . • Assume a model P = { f X ( x | θ ) : θ ∈ Ω ⊂ R p } where the functional form of f X ( x | θ ) is known, but θ is unknown. • Task is to use data x to make inference on θ

  2. X n is x n is called the estimate of • X • Suppose n • Define w • Define w . . x , and x i.i.d. , where X n . . . . X . X n n n i X i X . X X n X . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . . . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Point Estimation . Definition . w x . Example . . w x . The realization of the estimation, . called a point estimator of w X Then w X 7 / 24 . . . . . . . . . . . . . . . . . . If we use a function of sample w ( X 1 , · · · , X n ) as a ”guess” of τ ( θ ) , where τ ( θ ) is a function of true parameter θ .

  3. x n is called the estimate of • X • Suppose n • Define w • Define w . X . . X n i.i.d. , where . , and x x . X n . n n i X i X . X X n X . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . . . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Point Estimation . Definition . . The realization of the estimation, w x w x . . Example . . 7 / 24 . . . . . . . . . . . . . . . . . . If we use a function of sample w ( X 1 , · · · , X n ) as a ”guess” of τ ( θ ) , where τ ( θ ) is a function of true parameter θ . Then w ( X ) = w ( X 1 , · · · , X n ) is called a point estimator of τ ( θ ) .

  4. • Suppose n • Define w • Define w . i , and x x . X X n n n X i . X . X X n X . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 i.i.d. . . Example . . . . . . . . Likelihood Function Method of Moments MLE . Summary Point Estimation . Definition . . . 7 / 24 . . . . . . . . . . . . . . . . . . If we use a function of sample w ( X 1 , · · · , X n ) as a ”guess” of τ ( θ ) , where τ ( θ ) is a function of true parameter θ . Then w ( X ) = w ( X 1 , · · · , X n ) is called a point estimator of τ ( θ ) . The realization of the estimation, w ( x ) = w ( x 1 , · · · , x n ) is called the estimate of τ ( θ ) . • X 1 , · · · , X n ∼ N ( θ, 1) , where θ ∈ Ω ∈ R .

  5. • Define w • Define w . X i . i.i.d. X X n n n i X . . X X n X . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . Example . Point Estimation . . . . . . . . Likelihood Function Method of Moments MLE . Summary . Definition . . 7 / 24 . . . . . . . . . . . . . . . . . . If we use a function of sample w ( X 1 , · · · , X n ) as a ”guess” of τ ( θ ) , where τ ( θ ) is a function of true parameter θ . Then w ( X ) = w ( X 1 , · · · , X n ) is called a point estimator of τ ( θ ) . The realization of the estimation, w ( x ) = w ( x 1 , · · · , x n ) is called the estimate of τ ( θ ) . • X 1 , · · · , X n ∼ N ( θ, 1) , where θ ∈ Ω ∈ R . • Suppose n = 6 , and ( x 1 , · · · , x 6 ) = (2 . 0 , 2 . 1 , 2 . 9 , 2 . 6 , 1 . 2 , 1 . 8) .

  6. • Define w . . Example . . i.i.d. n X . X n X . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 7 / 24 . Likelihood Function Method of Moments . . . . . . . MLE . . Summary Point Estimation . Definition . . . . . . . . . . . . . . . . . . If we use a function of sample w ( X 1 , · · · , X n ) as a ”guess” of τ ( θ ) , where τ ( θ ) is a function of true parameter θ . Then w ( X ) = w ( X 1 , · · · , X n ) is called a point estimator of τ ( θ ) . The realization of the estimation, w ( x ) = w ( x 1 , · · · , x n ) is called the estimate of τ ( θ ) . • X 1 , · · · , X n ∼ N ( θ, 1) , where θ ∈ Ω ∈ R . • Suppose n = 6 , and ( x 1 , · · · , x 6 ) = (2 . 0 , 2 . 1 , 2 . 9 , 2 . 6 , 1 . 2 , 1 . 8) . • Define w 1 ( X 1 , · · · , X n ) = 1 ∑ n i =1 X i = X = 2 . 1 .

  7. . Definition February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang n i.i.d. . . Example . . . . 7 / 24 . . Summary . MLE . . Method of Moments Likelihood Function Point Estimation . . . . . . . . . . . . . . . . . . . . . . . If we use a function of sample w ( X 1 , · · · , X n ) as a ”guess” of τ ( θ ) , where τ ( θ ) is a function of true parameter θ . Then w ( X ) = w ( X 1 , · · · , X n ) is called a point estimator of τ ( θ ) . The realization of the estimation, w ( x ) = w ( x 1 , · · · , x n ) is called the estimate of τ ( θ ) . • X 1 , · · · , X n ∼ N ( θ, 1) , where θ ∈ Ω ∈ R . • Suppose n = 6 , and ( x 1 , · · · , x 6 ) = (2 . 0 , 2 . 1 , 2 . 9 , 2 . 6 , 1 . 2 , 1 . 8) . • Define w 1 ( X 1 , · · · , X n ) = 1 ∑ n i =1 X i = X = 2 . 1 . • Define w 2 ( X 1 , · · · , X n ) = X (1) = 1 . 2 .

  8. . is obtained by solving equations like this. n n i X i E X . . . . . . Point estimator of T m E X m . . . . . . m k k Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 m X i . i . . . . . . . . Likelihood Function Method of Moments MLE . Summary Method of Moments A method to equate sample moments to population moments and solve equations. Sample moments Population moments m n n i X i E X m n n 8 / 24 . . . . . . . . . . . . . . . . . .

  9. . m n i . . . . . . Point estimator of T is obtained by solving equations like this. m i . . . . . . m k k Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 8 / 24 n MLE n equations. A method to equate sample moments to population moments and solve Method of Moments Summary . Method of Moments Population moments Likelihood Function . . . . . . . . Sample moments . . . . . . . . . . . . . . . . . . m 1 = 1 ∑ n µ ′ 1 = E [ X | θ ] = µ ′ 1 ( θ ) i =1 X i m 2 = 1 i =1 X 2 ∑ n µ ′ 2 = E [ X | θ ] = µ ′ 2 ( θ ) m 3 = 1 i =1 X 3 ∑ n µ ′ 3 = E [ X | θ ] = µ ′ 3 ( θ )

  10. . m . n i . . . . . . m . n . . . . . m k k Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 i 8 / 24 Population moments Summary . A method to equate sample moments to population moments and solve equations. Sample moments n MLE Method of Moments Method of Moments Likelihood Function . . . . . . . . . . . . . . . . . . . . . . . . . . m 1 = 1 ∑ n µ ′ 1 = E [ X | θ ] = µ ′ 1 ( θ ) i =1 X i m 2 = 1 i =1 X 2 ∑ n µ ′ 2 = E [ X | θ ] = µ ′ 2 ( θ ) m 3 = 1 i =1 X 3 ∑ n µ ′ 3 = E [ X | θ ] = µ ′ 3 ( θ ) Point estimator of T ( θ ) is obtained by solving equations like this.

  11. . . . i . . . . . . . n . . . . m k Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 i n 8 / 24 Population moments . . . . . . . . Likelihood Function Method of Moments MLE . Method of Moments A method to equate sample moments to population moments and solve equations. Sample moments Summary n . . . . . . . . . . . . . . . . . . m 1 = 1 ∑ n µ ′ 1 = E [ X | θ ] = µ ′ 1 ( θ ) i =1 X i m 2 = 1 i =1 X 2 ∑ n µ ′ 2 = E [ X | θ ] = µ ′ 2 ( θ ) m 3 = 1 i =1 X 3 ∑ n µ ′ 3 = E [ X | θ ] = µ ′ 3 ( θ ) Point estimator of T ( θ ) is obtained by solving equations like this. µ ′ = 1 ( θ ) m 1 µ ′ = 2 ( θ ) m 2 µ ′ = k ( θ )

  12. . n . E X X E X E X Var X n n i X i X n . i X i Solving the two equations above, X , n i X i X n . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . . . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Examples of method of moments estimator . Problem . . i.i.d. . Solution . . . 9 / 24 . . . . . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) . Find estimator for µ, σ 2 . X 1 , · · · , X n

  13. . i E X Var X n n i X i X n n X i . Solving the two equations above, X , n i X i X n . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 E X 9 / 24 . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Examples of method of moments estimator . Problem . . i.i.d. . Solution . . . . . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) . Find estimator for µ, σ 2 . X 1 , · · · , X n µ ′ = E X = µ = X 1

  14. . X i . n n i X n n i Solving the two equations above, . X , n i X i X n . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 9 / 24 Solution . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Examples of method of moments estimator . Problem . i.i.d. . . . . . . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) . Find estimator for µ, σ 2 . X 1 , · · · , X n µ ′ = E X = µ = X 1 E X 2 = [ E X ] 2 + Var ( X ) = µ 2 + σ 2 = 1 ∑ X 2 µ ′ = 2 i =1

  15. . X i . . n n i n n i Solving the two equations above, Solution X , n i X i X n . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 9 / 24 . Likelihood Function Examples of method of moments estimator . MLE Problem Method of Moments . . . . . . . . . . . i.i.d. Summary . . . . . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) . Find estimator for µ, σ 2 . X 1 , · · · , X n µ ′ = E X = µ = X 1 E X 2 = [ E X ] 2 + Var ( X ) = µ 2 + σ 2 = 1 ∑ X 2 µ ′ = 2 i =1 { ˆ µ = X

  16. . Solving the two equations above, . . . n n i n i X , . n i X i X n . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 Solution 9 / 24 . Problem Method of Moments Summary Likelihood Function . . . . i.i.d. . Examples of method of moments estimator . MLE . . . . . . . . . . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) . Find estimator for µ, σ 2 . X 1 , · · · , X n µ ′ = E X = µ = X 1 E X 2 = [ E X ] 2 + Var ( X ) = µ 2 + σ 2 = 1 ∑ X 2 µ ′ = 2 i =1 { ˆ µ = X µ 2 + ˆ σ 2 = 1 i =1 X 2 ∑ n ˆ

  17. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang i n i n n . . . Solution . i.i.d. 9 / 24 . . Method of Moments . Likelihood Function Examples of method of moments estimator Summary . MLE Problem . . . . . . . . . . . . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) . Find estimator for µ, σ 2 . X 1 , · · · , X n µ ′ = E X = µ = X 1 E X 2 = [ E X ] 2 + Var ( X ) = µ 2 + σ 2 = 1 ∑ X 2 µ ′ = 2 i =1 { ˆ µ = X µ 2 + ˆ σ 2 = 1 i =1 X 2 ∑ n ˆ µ = X , ˆ σ 2 = ∑ n i =1 ( X i − X ) 2 / n . Solving the two equations above, ˆ

  18. p k f X x k p x p x E X k x x k Equating first two sample moments, n n i X i x . . n n i X i E X E X Var X k p kp p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 kp . . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Method of moments estimator - Binomial . Problem . . i.i.d. . Solution . . . . . 10 / 24 . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Binomial ( k , p ) . Find an estimator for k , p .

  19. . n Equating first two sample moments, n n i X i x E X kp n i . X i E X E X Var X k p kp p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 x 10 / 24 . Problem . . . . . . . . Likelihood Function Method of Moments MLE . Summary Method of moments estimator - Binomial . . . i.i.d. . Solution . . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Binomial ( k , p ) . Find an estimator for k , p . ( k ) f X ( x | k , p ) = p x (1 − p ) k − x x ∈ { 0 , 1 , · · · , k }

  20. . i . x Equating first two sample moments, n n X i n n X i . E X E X Var X k p kp p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 10 / 24 Solution . Method of moments estimator - Binomial MLE . . Method of Moments Problem . Likelihood Function . . . . . . . i.i.d. . . Summary . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Binomial ( k , p ) . Find an estimator for k , p . ( k ) f X ( x | k , p ) = p x (1 − p ) k − x x ∈ { 0 , 1 , · · · , k } 1 ∑ x = µ ′ = 1 = E X = kp i =1

  21. . n Solution . . . x Equating first two sample moments, n i.i.d. X i n n i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 10 / 24 Summary Likelihood Function Method of Moments . Problem . Method of moments estimator - Binomial . . . . . . . . . . MLE . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Binomial ( k , p ) . Find an estimator for k , p . ( k ) f X ( x | k , p ) = p x (1 − p ) k − x x ∈ { 0 , 1 , · · · , k } 1 ∑ x = µ ′ = 1 = E X = kp i =1 1 2 = E [ X 2 ] = ( E X ) 2 + Var ( X ) = k 2 p 2 + kp (1 − p ) ∑ X 2 µ ′ = i =1

  22. . The method of moments estimators are February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang of k and p . These are not the best estimators. It is possible to get negative estimates k X p n X . k Method of moments estimator - Binomial (cont’d) Summary . . . . . . . . Likelihood Function Method of Moments 11 / 24 MLE . . . . . . . . . . . . . . . . . . . 2 ˆ = X − 1 ∑ n i =1 ( X i − X ) 2

  23. . The method of moments estimators are February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang of k and p . These are not the best estimators. It is possible to get negative estimates k X p n X . k 11 / 24 Method of moments estimator - Binomial (cont’d) Likelihood Function . . . . . . . . Method of Moments MLE . Summary . . . . . . . . . . . . . . . . . . 2 ˆ = X − 1 ∑ n i =1 ( X i − X ) 2 ˆ = ˆ

  24. . The method of moments estimators are February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang of k and p . These are not the best estimators. It is possible to get negative estimates k X p n X . k 11 / 24 Method of moments estimator - Binomial (cont’d) Likelihood Function . . . . . . . . Method of Moments MLE . Summary . . . . . . . . . . . . . . . . . . 2 ˆ = X − 1 ∑ n i =1 ( X i − X ) 2 ˆ = ˆ

  25. . E X p p r p p r X i m i n n m p p p m E X m p February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang p Xp p r m X X i i n n X r X i . i . . . . . . . . Likelihood Function Method of Moments MLE . Summary Examples of MoM estimator - Negative Binomial . Problem . . 12 / 24 n n . Solution . . . . i.i.d. . . . . m . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Negative Binomial ( r , p ) . Find estimator for ( r , p ) .

  26. . m n n i X i E X r p p r p p p m m p X n n i X i X r m p p Xp p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 m 12 / 24 . Problem . . . . . . . . Likelihood Function Method of Moments MLE . Summary Examples of MoM estimator - Negative Binomial . . . Solution n n . . i.i.d. . . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Negative Binomial ( r , p ) . Find estimator for ( r , p ) . 1 X i = E X = r (1 − p ) ∑ = m 1 i =1

  27. . n . n n p p m m m X n i n X i X r m p p Xp p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 n p 12 / 24 Summary . Problem . . . MLE i.i.d. Method of Moments Likelihood Function . . . . . Solution . . . Examples of MoM estimator - Negative Binomial . . . . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Negative Binomial ( r , p ) . Find estimator for ( r , p ) . 1 X i = E X = r (1 − p ) ∑ = m 1 i =1 ) 2 1 ( r (1 − p ) + r (1 − p ) i = E X 2 = ∑ X 2 = m 2 p 2 i =1

  28. . p . n n p . n n p X Solution n r m p p Xp p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 12 / 24 . Problem Method of Moments . Likelihood Function Summary . . . . . . . Examples of MoM estimator - Negative Binomial MLE . . . . i.i.d. . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Negative Binomial ( r , p ) . Find estimator for ( r , p ) . 1 X i = E X = r (1 − p ) ∑ = m 1 i =1 ) 2 1 ( r (1 − p ) + r (1 − p ) i = E X 2 = ∑ X 2 = m 2 p 2 i =1 m 1 ˆ = = m 2 − m 2 2 1 i =1 X 2 ∑ n i − X 1

  29. . p . . n n p n . p X . n r p p p Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 Solution n 12 / 24 Examples of MoM estimator - Negative Binomial . . . . . . . . Likelihood Function Method of Moments i.i.d. . Summary MLE Problem . . . . . . . . . . . . . . . . . . . . . X 1 , · · · , X n ∼ Negative Binomial ( r , p ) . Find estimator for ( r , p ) . 1 X i = E X = r (1 − p ) ∑ = m 1 i =1 ) 2 1 ( r (1 − p ) + r (1 − p ) i = E X 2 = ∑ X 2 = m 2 p 2 i =1 m 1 ˆ = = m 2 − m 2 2 1 i =1 X 2 ∑ n i − X 1 m 1 ˆ X ˆ ˆ = p = 1 − ˆ 1 − ˆ

  30. a i Y i follows . , in general, the distribution is hard to obtain. k i a i Y i , where a i s are known constants with n i a i r i It is often reasonable to assume that the distribution of . k i approximately. Find a moment-based estimator of . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 However, the distribution of r k , respectively. We know that the distribution 13 / 24 Summary . . . . . . . . Likelihood Function Method of Moments MLE . Satterthwaite Approximation . Problem . . . . . . . . . . . . . . . . . . . . Let Y 1 , · · · , Y k are independently (but not identically) distributed random variables from χ 2 r 1 , · · · , χ 2 ∑ n i =1 Y i is also chi-squared with degrees of freedom equal to ∑ k i =1 r i .

  31. a i Y i follows . Problem February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . approximately. Find a moment-based estimator of i k It is often reasonable to assume that the distribution of r k , respectively. We know that the distribution . . . 13 / 24 . Method of Moments Summary . . MLE . . . . . . . Likelihood Function Satterthwaite Approximation . . . . . . . . . . . . . . . . . . Let Y 1 , · · · , Y k are independently (but not identically) distributed random variables from χ 2 r 1 , · · · , χ 2 ∑ n i =1 Y i is also chi-squared with degrees of freedom equal to ∑ k i =1 r i . However, the distribution of ∑ k i =1 a i Y i , where a i s are known constants with ∑ n i =1 a i r i = 1 , in general, the distribution is hard to obtain.

  32. . Summary February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang r k , respectively. We know that the distribution . . . Problem . Satterthwaite Approximation 13 / 24 . MLE Method of Moments Likelihood Function . . . . . . . . . . . . . . . . . . . . . . . . . . Let Y 1 , · · · , Y k are independently (but not identically) distributed random variables from χ 2 r 1 , · · · , χ 2 ∑ n i =1 Y i is also chi-squared with degrees of freedom equal to ∑ k i =1 r i . However, the distribution of ∑ k i =1 a i Y i , where a i s are known constants with ∑ n i =1 a i r i = 1 , in general, the distribution is hard to obtain. It is often reasonable to assume that the distribution of ∑ k i =1 a i Y i follows 1 ν χ 2 ν approximately. Find a moment-based estimator of ν .

  33. . E X i a i r i E X To match the second moments, E k i a i Y i Therefore, the method of moment estimator of a i EY i is k i a i Y i Note that can be negative, which is not desirable. Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 k i . k . . . . . . . . Likelihood Function Method of Moments MLE . Summary A Naive Solution E k i a i Y i 14 / 24 . . . . . . . . . . . . . . . . . . To match the first moment, let X ∼ χ 2 ν / ν . Then E ( X ) = 1 , and Var ( X ) = 2/ ν .

  34. . Therefore, the method of moment estimator of k To match the second moments, E k i a i Y i E X is a i Y i k i a i Y i Note that can be negative, which is not desirable. Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . k 14 / 24 A Naive Solution . . . . . . . . Likelihood Function E Method of Moments MLE . Summary . . . . . . . . . . . . . . . . . . To match the first moment, let X ∼ χ 2 ν / ν . Then E ( X ) = 1 , and Var ( X ) = 2/ ν . ( k ) ∑ ∑ ∑ = a i EY i = a i r i = 1 = E ( X ) i =1 i =1 i =1

  35. . is . k To match the second moments, E a i Y i E Therefore, the method of moment estimator of k a i Y i i a i Y i Note that can be negative, which is not desirable. Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 k 14 / 24 Summary . Method of Moments A Naive Solution Likelihood Function . . . . . MLE . . E . . . . . . . . . . . . . . . . . . . To match the first moment, let X ∼ χ 2 ν / ν . Then E ( X ) = 1 , and Var ( X ) = 2/ ν . ( k ) ∑ ∑ ∑ = a i EY i = a i r i = 1 = E ( X ) i =1 i =1 i =1 ) 2 ( k = 2 ∑ X 2 ) ( = ν + 1 i =1

  36. . A Naive Solution February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang E a i Y i E To match the second moments, . k a i Y i E k Summary Method of Moments . . . . . . . . Likelihood Function 14 / 24 . MLE . . . . . . . . . . . . . . . . . . To match the first moment, let X ∼ χ 2 ν / ν . Then E ( X ) = 1 , and Var ( X ) = 2/ ν . ( k ) ∑ ∑ ∑ = a i EY i = a i r i = 1 = E ( X ) i =1 i =1 i =1 ) 2 ( k = 2 ∑ X 2 ) ( = ν + 1 i =1 Therefore, the method of moment estimator of ν is 2 ˆ ν = i =1 a i Y i ) 2 − 1 ( ∑ k Note that ν can be negative, which is not desirable.

  37. . a i Y i k i a i Y i Var k i a i Y i E k i a i Y i Var k i E . k i a i Y i E k i a i Y i Var k i a i Y i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 E 15 / 24 Method of Moments Var MLE . Summary An alternative Solution To match the second moments, E Likelihood Function . . . . a i Y i . . a i Y i . . k . . . . . . . . . . . . . . . . . . ) 2 ] 2 ( k ( k ) [ ∑ ∑ ∑ = + E ( a i Y i ) i =1 i =1 i =1

  38. . a i Y i . k Var k i a i Y i E k i E a i Y i k i a i Y i Var k i a i Y i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 k 15 / 24 . Likelihood Function Method of Moments . MLE . . . a i Y i . . . . E To match the second moments, Var An alternative Solution Summary . . . . . . . . . . . . . . . . . . ) 2 ] 2 ( k ( k ) [ ∑ ∑ ∑ = + E ( a i Y i ) i =1 i =1 i =1 ] 2   [  Var ( ∑ k i =1 a i Y i ) ∑ = E ( a i Y i ) ] 2 + 1   [  E ( ∑ k i =1 a i Y i ) i =1

  39. . a i Y i a i Y i k . E k i Var a i Y i k i a i Y i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 Var k 15 / 24 To match the second moments, MLE Method of Moments Likelihood Function An alternative Solution . . . . Summary . . . E . . . . . . . . . . . . . . . . . . . . ) 2 ] 2 ( k ( k ) [ ∑ ∑ ∑ = + E ( a i Y i ) i =1 i =1 i =1 ] 2   [  Var ( ∑ k i =1 a i Y i ) ∑ = E ( a i Y i ) ] 2 + 1   [  E ( ∑ k i =1 a i Y i ) i =1    Var ( ∑ k i =1 a i Y i )  = 2 = ] 2 + 1 ν + 1   [ E ( ∑ k i =1 a i Y i )

  40. . An alternative Solution February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . k a i Y i Var a i Y i E To match the second moments, k Summary Likelihood Function . . . . . . . . . 15 / 24 MLE Method of Moments . . . . . . . . . . . . . . . . . . ) 2 ] 2 ( k ( k ) [ ∑ ∑ ∑ = + E ( a i Y i ) i =1 i =1 i =1 ] 2   [  Var ( ∑ k i =1 a i Y i ) ∑ = E ( a i Y i ) ] 2 + 1   [  E ( ∑ k i =1 a i Y i ) i =1    Var ( ∑ k i =1 a i Y i )  = 2 = ] 2 + 1 ν + 1   [ E ( ∑ k i =1 a i Y i ) ] 2 [ E ( ∑ k 2 i =1 a i Y i ) = ν Var ( ∑ k i =1 a i Y i )

  41. a i EY i r i Y i . n a i Var Y i n i r i Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator i k a i Y i n i a i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 i a i Y i . i . . . . . . . . Likelihood Function Method of Moments 16 / 24 MLE . Summary Alternative Solution (cont’d) independent chi-squared random variables. Var n . . . . . . . . . . . . . . . . . . To match the second moments, Finally, use the fact that Y 1 , · · · , Y k are

  42. a i EY i r i Y i . n n i r i Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator a i Y i i . n i a i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 k 16 / 24 . . . . . . Likelihood Function Method of Moments MLE Summary Alternative Solution (cont’d) independent chi-squared random variables. . . n . . . . . . . . . . . . . . . . . . . To match the second moments, Finally, use the fact that Y 1 , · · · , Y k are ∑ ∑ Var ( a i Y i ) = a i Var ( Y i ) i =1 i =1

  43. r i Y i . i n r i Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator n n a i Y i k i a i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 16 / 24 n . . . . . MLE Method of Moments Summary Likelihood Function independent chi-squared random variables. . . . . Alternative Solution (cont’d) . . . . . . . . . . . . . . . . . . To match the second moments, Finally, use the fact that Y 1 , · · · , Y k are ∑ ∑ Var ( a i Y i ) = a i Var ( Y i ) i =1 i =1 a 2 i ( EY i ) 2 ∑ = 2 i =1

  44. r i Y i . i n r i Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator n n a i Y i k i a i Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 16 / 24 n . . . . . MLE Method of Moments Summary Likelihood Function independent chi-squared random variables. . . . . Alternative Solution (cont’d) . . . . . . . . . . . . . . . . . . To match the second moments, Finally, use the fact that Y 1 , · · · , Y k are ∑ ∑ Var ( a i Y i ) = a i Var ( Y i ) i =1 i =1 a 2 i ( EY i ) 2 ∑ = 2 i =1

  45. . independent chi-squared random variables. February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang i i we obtain Satterthwaite’s estimator Substituting this expression for the variance and removing expectations, r i n . k n 16 / 24 Alternative Solution (cont’d) . Summary Likelihood Function . . . . . . Method of Moments . MLE . . . . . . . . . . . . . . . . . . . To match the second moments, Finally, use the fact that Y 1 , · · · , Y k are ∑ ∑ Var ( a i Y i ) = a i Var ( Y i ) i =1 i =1 a 2 i ( EY i ) 2 ∑ = 2 i =1 ∑ n i =1 a i Y i ˆ = ν a 2 r i Y 2 ∑ n i =1

  46. • let • L • More formally, L • and February 7th, 2013 x x attains its maximum. x x L x where . Biostatistics 602 - Lecture 09 x be the value such that x is called the maximum likelihood estimate of based on data x , X is the maximum likelihood estimator (MLE) of . Hyun Min Kang • . . Method of Moments . . . . . . . . Likelihood Function 17 / 24 MLE . Summary Maximum Likelihood Estimator . Definition . . . . . . . . . . . . . . . . . . . . • For a given sample point x = ( x 1 , · · · , x n ) ,

  47. • L • More formally, L • and . . x x L x where x x is called the maximum likelihood estimate of • based on data x , X is the maximum likelihood estimator (MLE) of . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 x attains its maximum. 17 / 24 . Method of Moments . . . . . . . . Likelihood Function MLE . Summary Maximum Likelihood Estimator . Definition . . . . . . . . . . . . . . . . . . . . • For a given sample point x = ( x 1 , · · · , x n ) , • let ˆ θ ( x ) be the value such that

  48. • More formally, L • and • x x L x where x . . . based on data x , X is the maximum likelihood estimator (MLE) of . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 x is called the maximum likelihood estimate of 17 / 24 MLE . . . . . Likelihood Function Method of Moments . . Summary Maximum Likelihood Estimator . Definition . . . . . . . . . . . . . . . . . . . . . . • For a given sample point x = ( x 1 , · · · , x n ) , • let ˆ θ ( x ) be the value such that • L ( θ | x ) attains its maximum.

  49. • and . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . X is the maximum likelihood estimator (MLE) of based on data x , x is called the maximum likelihood estimate of • . . Definition . Maximum Likelihood Estimator Summary . . . . . . . . Likelihood Function Method of Moments 17 / 24 MLE . . . . . . . . . . . . . . . . . . . • For a given sample point x = ( x 1 , · · · , x n ) , • let ˆ θ ( x ) be the value such that • L ( θ | x ) attains its maximum. • More formally, L (ˆ θ ( x ) | x ) ≥ L ( θ | x ) ∀ θ ∈ Ω where ˆ θ ( x ) ∈ Ω .

  50. • and . Maximum Likelihood Estimator February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . X is the maximum likelihood estimator (MLE) of . . Definition . . Summary . . . . . . . . . Likelihood Function 17 / 24 Method of Moments MLE . . . . . . . . . . . . . . . . . . • For a given sample point x = ( x 1 , · · · , x n ) , • let ˆ θ ( x ) be the value such that • L ( θ | x ) attains its maximum. • More formally, L (ˆ θ ( x ) | x ) ≥ L ( θ | x ) ∀ θ ∈ Ω where ˆ θ ( x ) ∈ Ω . • ˆ θ ( x ) is called the maximum likelihood estimate of θ based on data x ,

  51. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . . Definition . Maximum Likelihood Estimator Summary . MLE . . . . 17 / 24 Likelihood Function . . . . Method of Moments . . . . . . . . . . . . . . . . . . • For a given sample point x = ( x 1 , · · · , x n ) , • let ˆ θ ( x ) be the value such that • L ( θ | x ) attains its maximum. • More formally, L (ˆ θ ( x ) | x ) ≥ L ( θ | x ) ∀ θ ∈ Ω where ˆ θ ( x ) ∈ Ω . • ˆ θ ( x ) is called the maximum likelihood estimate of θ based on data x , • and ˆ θ ( X ) is the maximum likelihood estimator (MLE) of θ .

  52. f X x f X x i n exp . n . . . L x n i e i . x i n i x i where . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . . . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary Example of MLE - Exponential Distribution . Problem . . i.i.d. . Solution . 18 / 24 . . . . . . . . . . . . . . . . . . Let X 1 , · · · , X n ∼ Exponential ( β ) . Find MLE of β .

  53. n exp . . . n n i e x i n . i x i where . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . Solution i.i.d. Summary . . . . . . . . Likelihood Function Method of Moments MLE . Example of MLE - Exponential Distribution Problem . . . 18 / 24 . . . . . . . . . . . . . . . . . . Let X 1 , · · · , X n ∼ Exponential ( β ) . Find MLE of β . ∏ L ( β | x ) = f X ( x | θ ) = f X ( x i | θ ) i =1

  54. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang x i n n n . . . Solution . i.i.d. . 18 / 24 Problem . Likelihood Function . . . . Method of Moments . . . . MLE . Summary Example of MLE - Exponential Distribution . . . . . . . . . . . . . . . . . . Let X 1 , · · · , X n ∼ Exponential ( β ) . Find MLE of β . ∏ L ( β | x ) = f X ( x | θ ) = f X ( x i | θ ) i =1 [ 1 ( ) ] = 1 ∏ β e − x i / β ∑ = − β n exp β i =1 i =1 where β > 0 .

  55. n exp . n i x i n log l n i x i n i x i x i n n i x i n x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 n i . n . . . . . . . . Likelihood Function Method of Moments MLE . Summary Use the derivative to find potential MLE log-likelihood function l x log L x log 19 / 24 . . . . . . . . . . . . . . . . . . To maximize the likelihood function L ( β | x ) is equivalent to maximize the

  56. . n n i x i n log l n i x i n i . x i n n i x i n x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 x i 19 / 24 n Summary . . . . . . . . Likelihood Function Method of Moments MLE . Use the derivative to find potential MLE log-likelihood function . . . . . . . . . . . . . . . . . . To maximize the likelihood function L ( β | x ) is equivalent to maximize the [ ( )] 1 ∑ l ( β | x ) log L ( β | x ) = log − = β n exp β i =1

  57. . x i x i l n i x i n n i n n n i x i n x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 19 / 24 . . . . Likelihood Function MLE . Summary Use the derivative to find potential MLE log-likelihood function . . . . Method of Moments . . . . . . . . . . . . . . . . . . To maximize the likelihood function L ( β | x ) is equivalent to maximize the [ ( )] 1 ∑ l ( β | x ) log L ( β | x ) = log − = β n exp β i =1 ∑ n i =1 x i = − − n log β β

  58. . n February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang x n x i i n n x i i n . x i 19 / 24 . Summary MLE Method of Moments Likelihood Function . . . . . . . Use the derivative to find potential MLE . log-likelihood function . . . . . . . . . . . . . . . . . . To maximize the likelihood function L ( β | x ) is equivalent to maximize the [ ( )] 1 ∑ l ( β | x ) log L ( β | x ) = log − = β n exp β i =1 ∑ n i =1 x i = − − n log β β ∑ n ∂ l i =1 x i − n = β = 0 β 2 ∂β

  59. . log-likelihood function February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang x n x i i n x i n . x i n 19 / 24 MLE . . . . . . . . Use the derivative to find potential MLE Likelihood Function Summary Method of Moments . . . . . . . . . . . . . . . . . . . To maximize the likelihood function L ( β | x ) is equivalent to maximize the [ ( )] 1 ∑ l ( β | x ) log L ( β | x ) = log − = β n exp β i =1 ∑ n i =1 x i = − − n log β β ∑ n ∂ l i =1 x i − n = β = 0 β 2 ∂β ∑ = n β i =1

  60. . Use the derivative to find potential MLE February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang n x i n . x i n log-likelihood function 19 / 24 Summary Likelihood Function . . . . . Method of Moments . . . . MLE . . . . . . . . . . . . . . . . . . To maximize the likelihood function L ( β | x ) is equivalent to maximize the [ ( )] 1 ∑ l ( β | x ) log L ( β | x ) = log − = β n exp β i =1 ∑ n i =1 x i = − − n log β β ∑ n ∂ l i =1 x i − n = β = 0 β 2 ∂β ∑ = n β i =1 ∑ n ˆ i =1 x i β = = x

  61. . n i x i n x x nx x x . n Therefore, we can conclude that X X is unique local maximum on the interval Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 n 20 / 24 Likelihood Function Method of Moments MLE . Summary Use the double derivative to confirm local maximum . . . . . . . . . . . . . . . . . . . . . . . . . . ∂ 2 l � ∑ n � � i =1 x i � − 2 = + n � � ∂β 2 β 3 β 2 � � β = x β = x

  62. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang the interval X is unique local maximum on X Therefore, we can conclude that n x n x nx x 20 / 24 Summary . . MLE Method of Moments Likelihood Function . . . . Use the double derivative to confirm local maximum . . . . . . . . . . . . . . . . . . . . . ∂ 2 l � ∑ n � � i =1 x i � − 2 = + n � � ∂β 2 β 3 β 2 � � β = x β = x ( − 2 ∑ n )� 1 i =1 x i � = + n � β 2 β � β = x

  63. . Use the double derivative to confirm local maximum February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang the interval X is unique local maximum on X Therefore, we can conclude that n x x . 20 / 24 Summary MLE Method of Moments Likelihood Function . . . . . . . . . . . . . . . . . . . . . . . . . . . ∂ 2 l � ∑ n � � i =1 x i � − 2 = + n � � ∂β 2 β 3 β 2 � � β = x β = x ( − 2 ∑ n )� 1 i =1 x i � = + n � β 2 β � β = x 1 ( − 2 nx ) = + n x 2

  64. . Summary February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang the interval X is unique local maximum on X Therefore, we can conclude that x . Use the double derivative to confirm local maximum 20 / 24 . Method of Moments Likelihood Function MLE . . . . . . . . . . . . . . . . . . . . . . . . . . ∂ 2 l � ∑ n � � i =1 x i � − 2 = + n � � ∂β 2 β 3 β 2 � � β = x β = x ( − 2 ∑ n )� 1 i =1 x i � = + n � β 2 β � β = x 1 ( − 2 nx ) = + n x 2 1 = x 2 ( − n ) < 0

  65. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang the interval x . Use the double derivative to confirm local maximum Summary 20 / 24 MLE . . . . Likelihood Function . . . . Method of Moments . . . . . . . . . . . . . . . . . . ∂ 2 l � ∑ n � � i =1 x i � − 2 = + n � � ∂β 2 β 3 β 2 � � β = x β = x ( − 2 ∑ n )� 1 i =1 x i � = + n � β 2 β � β = x 1 ( − 2 nx ) = + n x 2 1 = x 2 ( − n ) < 0 Therefore, we can conclude that ˆ β ( X ) = X is unique local maximum on

  66. . x i x l x n i x i n log n i n , use log x n i x i n L x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 lim If . x . . . . . . . . Likelihood Function Method of Moments MLE . Summary Check boundary and confirm global maximum l x n i x i n log L 21 / 24 . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞

  67. . i lim x l x n i x i n log n x i If n n i x i n L x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 , use log x x . L . . . . . . . . Likelihood Function Method of Moments MLE . Summary Check boundary and confirm global maximum 21 / 24 . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β

  68. . i lim x l x n i x i n log n x i If n n i x i n L x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 , use log x 21 / 24 . . . . . . . . . . Likelihood Function Method of Moments MLE Summary Check boundary and confirm global maximum . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β L ( β | x ) → 0

  69. . n x n i x i n log n i x i n . i x i n L x Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 l 21 / 24 Summary Check boundary and confirm global maximum . . . . Method of Moments MLE . Likelihood Function . . . . . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β L ( β | x ) → 0 β ( x β − 1) If β → 0 , use log ( x ) = lim β → 0 1

  70. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang x L n x i i n n x i i n 21 / 24 . . Summary Check boundary and confirm global maximum Method of Moments Likelihood Function . . . . . MLE . . . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β L ( β | x ) → 0 β ( x β − 1) If β → 0 , use log ( x ) = lim β → 0 1 ∑ n i =1 x i l ( β | x ) = − − n log β β

  71. . Check boundary and confirm global maximum February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang x L n x i i n . 21 / 24 Summary . . Likelihood Function . . . . . . . MLE Method of Moments . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β L ( β | x ) → 0 β ( x β − 1) If β → 0 , use log ( x ) = lim β → 0 1 ∑ n i =1 x i l ( β | x ) = − − n log β β ( 1 ∑ n ) β β β − 1 i =1 x i = − − n β

  72. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang x L . Check boundary and confirm global maximum Summary 21 / 24 MLE . . . . . . . Likelihood Function . Method of Moments . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β L ( β | x ) → 0 β ( x β − 1) If β → 0 , use log ( x ) = lim β → 0 1 ∑ n i =1 x i l ( β | x ) = − − n log β β ( 1 ∑ n ) β β β − 1 i =1 x i = − − n β i =1 x i − n ( β β − 1) ∑ n = − → −∞ β

  73. . MLE February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . Check boundary and confirm global maximum Summary . 21 / 24 Method of Moments . Likelihood Function . . . . . . . . . . . . . . . . . . . . . . . . . β ∈ (0 , ∞ ) . If β → ∞ ∑ n i =1 x i l ( β | x ) = − − n log β → −∞ β L ( β | x ) → 0 β ( x β − 1) If β → 0 , use log ( x ) = lim β → 0 1 ∑ n i =1 x i l ( β | x ) = − − n log β β ( 1 ∑ n ) β β β − 1 i =1 x i = − − n β i =1 x i − n ( β β − 1) ∑ n = − → −∞ β L ( β | x ) → 0

  74. . Therefore l at x . 3 L x (lowest bound) when approaches the boundary x and L 2 x attains the global maximum when x X X is the MLE of . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 l . . Method of Moments . . . . . . . . Likelihood Function MLE . Summary Putting Things Together . . 1 22 / 24 . . . . . . . . . . . . . . . . . . ∂β = 0 at ˆ ∂ l β = x

  75. . x and L 2 . 3 L x (lowest bound) when approaches the boundary Therefore l x attains the global maximum when . x X X is the MLE of . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . . 22 / 24 MLE . . . . . . . . Likelihood Function Method of Moments . Summary Putting Things Together . . 1 . . . . . . . . . . . . . . . . . . ∂β = 0 at ˆ ∂ l β = x ∂β 2 < 0 at ˆ ∂ 2 l β = x

  76. . x attains the global maximum when . 2 . . Therefore l x and L x . X X is the MLE of . Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 . 22 / 24 . . . . . Likelihood Function Method of Moments MLE . . Summary . . Putting Things Together . . 1 . . . . . . . . . . . . . . . . . . ∂β = 0 at ˆ ∂ l β = x ∂β 2 < 0 at ˆ ∂ 2 l β = x 3 L ( β | x ) → 0 (lowest bound) when β approaches the boundary

  77. . . February 7th, 2013 Biostatistics 602 - Lecture 09 Hyun Min Kang . . 2 . . . 1 . 22 / 24 Putting Things Together . . . . . MLE . . Method of Moments Likelihood Function . . Summary . . . . . . . . . . . . . . . . . . ∂β = 0 at ˆ ∂ l β = x ∂β 2 < 0 at ˆ ∂ 2 l β = x 3 L ( β | x ) → 0 (lowest bound) when β approaches the boundary Therefore l ( β | x ) and L ( β | x ) attains the global maximum when ˆ β = x ˆ β ( X ) = X is the MLE of β .

  78. • For one-dimensional parameter, negative second order derivative • For two-dimensional parameter, suppose L • Check boundary points to see whether boundary gives global maximum. • Use numerical methods • Or perform directly maximization, using inequalities, or properties of L is the likelihood function. Then we need to show (a) L or . . (b) Determinant of second-order derivative is positive If the function is NOT differentiable with respect to . the function. Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 implies local maximum. 2 Check second-order derivative to check local maximum. . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary How do we find MLE? . . 1 Find candidates that makes first order derivative to be zero . 23 / 24 . . . . . . . . . . . . . . . . . . If the function is differentiable with respect to θ .

  79. • For one-dimensional parameter, negative second order derivative • For two-dimensional parameter, suppose L • Check boundary points to see whether boundary gives global maximum. • Use numerical methods • Or perform directly maximization, using inequalities, or properties of L is the likelihood function. Then we need to show (a) L or . . (b) Determinant of second-order derivative is positive If the function is NOT differentiable with respect to . the function. Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 implies local maximum. 2 Check second-order derivative to check local maximum. . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary How do we find MLE? . . 1 Find candidates that makes first order derivative to be zero . 23 / 24 . . . . . . . . . . . . . . . . . . If the function is differentiable with respect to θ .

  80. • For one-dimensional parameter, negative second order derivative • For two-dimensional parameter, suppose L • Check boundary points to see whether boundary gives global maximum. • Use numerical methods • Or perform directly maximization, using inequalities, or properties of L is the likelihood function. Then we need to show (a) L or . . (b) Determinant of second-order derivative is positive If the function is NOT differentiable with respect to . the function. Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 implies local maximum. 2 Check second-order derivative to check local maximum. . . . . . . . . . . Likelihood Function Method of Moments MLE . Summary How do we find MLE? . . 1 Find candidates that makes first order derivative to be zero . 23 / 24 . . . . . . . . . . . . . . . . . . If the function is differentiable with respect to θ .

Recommend

Biostatistics Preparatory Course: Methods and Computing Lecture 9 Maximum Likelihood &amp; the

Biostatistics Preparatory Course: Methods and Computing Lecture 9 Maximum Likelihood &amp; the

Biostatistics Preparatory Course: Methods and Computing Lecture 9 Maximum Likelihood &amp; the Bootstrap Methods and Computing Harvard University Department of Biostatistics 1 / 16 Overview: Maximum Likelihood Estimation Consider estimating a

455 views • 18 slides
Pseudo-likelihood estimation for non hereditary Gibbs point processes Frdric Lavancier ,

Pseudo-likelihood estimation for non hereditary Gibbs point processes Frdric Lavancier ,

Introduction Definitions Campbell Estimation Simulations Pseudo-likelihood estimation for non hereditary Gibbs point processes Frdric Lavancier , Laboratoire Jean Leray, Nantes, France. Joint work with David Dereudre , LAMAV,

554 views • 42 slides
Kernel Recursive ABC: Point Estimation with Intractable Likelihood Motonobu Kanagawa EURECOM,

Kernel Recursive ABC: Point Estimation with Intractable Likelihood Motonobu Kanagawa EURECOM,

Kernel Recursive ABC: Point Estimation with Intractable Likelihood Motonobu Kanagawa EURECOM, Sophia Antipolis, France (Previously U. Tbingen) ISM-UUlm Workshop, October 2019 1 / 44 Contents of This Talk 1. Kernel Recursive ABC: Point

1.47k views • 131 slides
Lecture 8: Maximum Likelihood Estimation (MLE) (contd.) Maximum a posteriori (MAP)

Lecture 8: Maximum Likelihood Estimation (MLE) (contd.) Maximum a posteriori (MAP)

Lecture 8: Maximum Likelihood Estimation (MLE) (contd.) Maximum a posteriori (MAP) estimation Nave Bayes Classifier Aykut Erdem March 2016 Hacettepe University Last time Flipping a Coin I have a coin, if I flip it, whats the

932 views • 82 slides
ECON 626: Applied Microeconomics Lecture 11: Maximum Likelihood Estimation Professors: Pamela

ECON 626: Applied Microeconomics Lecture 11: Maximum Likelihood Estimation Professors: Pamela

ECON 626: Applied Microeconomics Lecture 11: Maximum Likelihood Estimation Professors: Pamela Jakiela and Owen Ozier Maximum Likelihood: Motivation So far, weve been thinking about average treatment effects, but the ATE may or may not be

564 views • 23 slides
Chapter 8: Estimation In this chapter we will cover: 1. The likelihood and maximum likelihood

Chapter 8: Estimation In this chapter we will cover: 1. The likelihood and maximum likelihood

Chapter 8: Estimation In this chapter we will cover: 1. The likelihood and maximum likelihood estimation ( 8.3, 8.5 Rice) 2. Introduction to decision theory( 15.115.3) The likelihood function We have seen many cases where we have

413 views • 10 slides
Lecture 4: Bayesian Decision Theory and Max Likelihood Estimation Dr. Chengjiang Long Computer

Lecture 4: Bayesian Decision Theory and Max Likelihood Estimation Dr. Chengjiang Long Computer

Lecture 4: Bayesian Decision Theory and Max Likelihood Estimation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture 4 January 30, 2018

855 views • 59 slides
Binary choice 3.3 Maximum likelihood estimation Michel Bierlaire Output of the estimation

Binary choice 3.3 Maximum likelihood estimation Michel Bierlaire Output of the estimation

Binary choice 3.3 Maximum likelihood estimation Michel Bierlaire Output of the estimation We explain here the various outputs from the maximum likelihood esti- mation procedure. Solution of the maximum likelihood estimation The main

170 views • 5 slides
Approximate Bayesian logistic regression via penalized likelihood estimation with data

Approximate Bayesian logistic regression via penalized likelihood estimation with data

Introduction Methods and formulas The penlogit command Example Conclusions Approximate Bayesian logistic regression via penalized likelihood estimation with data augmentation Andrea Discacciati Nicola Orsini Unit of Biostatistics and Unit of

676 views • 24 slides
Lecture 7: Maximum Likelihood Estimation (MLE) Maximum a Posteriori (MAP) Aykut Erdem

Lecture 7: Maximum Likelihood Estimation (MLE) Maximum a Posteriori (MAP) Aykut Erdem

Lecture 7: Maximum Likelihood Estimation (MLE) Maximum a Posteriori (MAP) Aykut Erdem October 2016 Hacettepe University Administrative Assignment 2 will be out on Thursday It is due November 10 (i.e. in 2 weeks) You will

831 views • 70 slides
Likelihood Ratio Test Lecture 19 Biostatistics 602 - Statistical Inference . . . . Unbiased

Likelihood Ratio Test Lecture 19 Biostatistics 602 - Statistical Inference . . . . Unbiased

. Summary March 26th, 2013 Biostatistics 602 - Lecture 19 Hyun Min Kang March 26th, 2013 Hyun Min Kang Likelihood Ratio Test Lecture 19 Biostatistics 602 - Statistical Inference . . . . Unbiased Test LRT Recap . . . . . . . . 1

1.76k views • 144 slides
Binary choice 3.3 Maximum likelihood estimation Michel Bierlaire Maximum likelihood

Binary choice 3.3 Maximum likelihood estimation Michel Bierlaire Maximum likelihood

Binary choice 3.3 Maximum likelihood estimation Michel Bierlaire Maximum likelihood estimation We now estimate the values of the unknown parameters 1 ,. . . , K from a sample of observations drawn at random from the population. Each

758 views • 8 slides
Maximum Likelihood Estimator Lecture 10 Biostatistics 602 - Statistical Inference . Summary .

Maximum Likelihood Estimator Lecture 10 Biostatistics 602 - Statistical Inference . Summary .

. . February 12th, 2013 Biostatistics 602 - Lecture 10 Hyun Min Kang February 12th, 2013 Hyun Min Kang Maximum Likelihood Estimator Lecture 10 Biostatistics 602 - Statistical Inference . Summary . . MLE Recap . . . . . . . . . 1

1.08k views • 84 slides
Point Estimation The goal of Point Estimation is to find the point in -space which gives the

Point Estimation The goal of Point Estimation is to find the point in -space which gives the

Point Estimation Point Estimation The goal of Point Estimation is to find the point in -space which gives the best estimate (measurement) of the parameter . We assume, as always, P (data | hypothesis) = P ( X | ) known. What we

711 views • 42 slides
Lecture 3, Estimation and model validation Erik Lindstrmm Maximum likelihood, recap (1)

Lecture 3, Estimation and model validation Erik Lindstrmm Maximum likelihood, recap (1)

Lecture 3, Estimation and model validation Erik Lindstrmm Maximum likelihood, recap (1) parametric class of models with known density some unknown parameter vector. defined as (2) argument, so this is equivalently written as Let x ( N

359 views • 34 slides
Maximum Likelihood Estimation CS 446 Maximum likelihood: abstract formulation Weve had one

Maximum Likelihood Estimation CS 446 Maximum likelihood: abstract formulation Weve had one

Maximum Likelihood Estimation CS 446 Maximum likelihood: abstract formulation Weve had one main meta-algorithm this semester: (Regularized) ERM principle: pick the model that minimizes an average loss over training data. 1 / 70

599 views • 42 slides
Maximum Likelihood Estimation CS 446 Maximum likelihood: abstract formulation Weve had one

Maximum Likelihood Estimation CS 446 Maximum likelihood: abstract formulation Weve had one

Maximum Likelihood Estimation CS 446 Maximum likelihood: abstract formulation Weve had one main meta-algorithm) this semester: (Regularized) ERM principle: pick the model that minimizes an average loss over training data. 1 / 76

643 views • 45 slides
Week 2: Maximum Likelihood Estimation Instructor: Sergey Levine 1 Recap: MLE for the binomial

Week 2: Maximum Likelihood Estimation Instructor: Sergey Levine 1 Recap: MLE for the binomial

Week 2: Maximum Likelihood Estimation Instructor: Sergey Levine 1 Recap: MLE for the binomial distribution In the previous lecture, we covered maximum likelihood estimation for the bi- nomial distribution. Lets recap the key ideas: What is

301 views • 5 slides
Lecture 4. Maximum Likelihood Estimation - confidence intervals. Igor Rychlik Chalmers

Lecture 4. Maximum Likelihood Estimation - confidence intervals. Igor Rychlik Chalmers

Lecture 4. Maximum Likelihood Estimation - confidence intervals. Igor Rychlik Chalmers Department of Mathematical Sciences Probability, Statistics and Risk, MVE300 Chalmers April 2013. Click on red text for extra material. Maximum

478 views • 18 slides
Lecture 3, Estimation and model validation Magnus Wiktorsson Maximum likelihood, recap argument,

Lecture 3, Estimation and model validation Magnus Wiktorsson Maximum likelihood, recap argument,

Lecture 3, Estimation and model validation Magnus Wiktorsson Maximum likelihood, recap argument, so this is equivalently written as parametric class of models with known density some unknown parameter vector. defined as Let x ( N ) = ( x 1

484 views • 34 slides
Lecture 7: Probability Review (contd.) Maximum Likelihood Estimation (MLE) Aykut Erdem

Lecture 7: Probability Review (contd.) Maximum Likelihood Estimation (MLE) Aykut Erdem

Lecture 7: Probability Review (contd.) Maximum Likelihood Estimation (MLE) Aykut Erdem November 2018 Hacettepe University Administrative Assignment 2 will be out tonight It is due November 24 (i.e. in 2 weeks) You will

685 views • 56 slides
Evaluation of Point Estimators Lecture 11 Biostatistics 602 - Statistical Inference . .

Evaluation of Point Estimators Lecture 11 Biostatistics 602 - Statistical Inference . .

. . February 14th, 2013 Biostatistics 602 - Lecture 11 Hyun Min Kang February 14th, 2013 Hyun Min Kang Evaluation of Point Estimators Lecture 11 Biostatistics 602 - Statistical Inference . . Summary Cramer-Rao . Evaluation MLE Recap

2.02k views • 136 slides
Maximum Likelihood Estimation MLE tool for parameter estimation good approach for cases

Maximum Likelihood Estimation MLE tool for parameter estimation good approach for cases

Maximum Likelihood Estimation MLE tool for parameter estimation good approach for cases when OLS (ordinary least squares) assumptions are violated e.g. for non-linear models with non-normal data in MLE, we estimate the parameters

1.01k views • 66 slides
15-388/688 - Practical Data Science: Maximum likelihood estimation, nave Bayes J. Zico Kolter

15-388/688 - Practical Data Science: Maximum likelihood estimation, nave Bayes J. Zico Kolter

15-388/688 - Practical Data Science: Maximum likelihood estimation, nave Bayes J. Zico Kolter Carnegie Mellon University Spring 2018 1 Outline Maximum likelihood estimation Naive Bayes Machine learning and maximum likelihood 2 Outline

352 views • 22 slides

More recommend