Applied Statistics Lecturer: Serena Arima Likelihood ML estimator - PowerPoint PPT Presentation

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Applied Statistics Lecturer: Serena Arima

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Statistical models Statistics concerns what can be learned from data using statistical models to study the variability of the data. The key feature of a statistical model is that variability is represented using probability distributions, which form the building-blocks from which the model is constructed. The key idea in statistical modelling is to treat the data as the outcome of a random experiment.

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Statistical models Statistics concerns what can be learned from data using statistical models to study the variability of the data. The key feature of a statistical model is that variability is represented using probability distributions, which form the building-blocks from which the model is constructed. The key idea in statistical modelling is to treat the data as the outcome of a random experiment.

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Statistical models Statistics concerns what can be learned from data using statistical models to study the variability of the data. The key feature of a statistical model is that variability is represented using probability distributions, which form the building-blocks from which the model is constructed. The key idea in statistical modelling is to treat the data as the outcome of a random experiment.

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Suppose we have observed the value y of a random variable Y whose probability density function is supposed known up to the value of a parameter θ . We write f ( y ; θ ) to emphasize that the density is function of both data y and parameter θ . θ ∈ Θ � parameter space Y ∈ Y � sample space

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Suppose we have observed the value y of a random variable Y whose probability density function is supposed known up to the value of a parameter θ . We write f ( y ; θ ) to emphasize that the density is function of both data y and parameter θ . θ ∈ Θ � parameter space Y ∈ Y � sample space

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Our goal is to make statements about the distribution of Y , based on the observed value y (make inference about θ ). A fundamental tool is the likelihood for θ based on y , which is defined as L ( θ ) = f ( y ; θ ) θ ∈ Θ regarded as a function of θ for fixed y . When Y is discrete we use f ( y ; θ ) = Pr ( Y = y ; θ ) .

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Binomial example : a coin gives Head with probability θ and Tail with probability 1 − θ . Suppose to toss the coin n = 10 times. 1 What is the parameter space? θ ∈ Θ = [ 0 , 1 ] 2 What is the sample space? (heads obtained tossing the coin 10 times) y ∈ { 0 , 1 , 2 , ..., 10 } 3 Which random variable does represent the experiment? Y ∼ Binomial ( 10 , θ )

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood The likelihood is � 10 � θ y ( 1 − θ ) 10 − y f ( y ; θ ) = y Suppose that the experiment leads to Y = y = 7: For θ = 0 . 6, L ( θ ; y ) = 0 . 215 Likelihood value for Bin(n=10,y=7) For θ = 0 . 5, L ( θ ; y ) = 0 . 117 Hence θ = 0 . 6 is more likely than 0.2 θ = 0 . 5; more precisely, θ = 0 . 6 is Likelihood 0 . 215 0.1 0 . 117 = 1 . 838 times more likely that θ = 0 . 5. 0.0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 What is the most likely value? Theta

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood When y = ( y 1 , y 2 , ..., y n ) is a collection of independent observations the likelihood is � n L ( θ ) = f ( y j ; θ ) . j = 1

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Normal example: we observe n replicates ( X 1 , X 2 , ..., X n ) from a Normal random variable X i ∼ N ( µ, σ 2 ) . The likelihood of a Normal experiment is � � 1 − n 2 σ 2 ( s 2 + (¯ L ( y ; θ = ( µ, σ 2 )) = x − µ ) 2 ) ( 2 π ) n / 2 ( σ 2 ) n / 2 exp Proof (blackboard).

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Suppose that n = 10 and σ 2 = 4 known and fixed. Hence L ( y ; θ = ( µ, σ 2 )) = L ( y , σ 2 ; θ = µ ) . For a sample z = ( 6 . 38 , 1 . 39 , 5 . 67 , 3 . 26 , 1 . 96 , 3 . 73 , − 0 . 32 , 0 . 54 , 2 . 53 , 5 . 40 ) the likelihood is Likelihood for sigma=4 fixed 8.33e−48 L ( µ ) 5e−48 1.66e−48 0 −2 −1 0 1 2 3 4 5 6 7 8 µ

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Suppose now that n = 10 and µ = 3 known and fixed. Hence L ( y ; θ = ( µ, σ 2 )) = L ( y , µ ; θ = σ 2 ) . For the same sample the likelihood is Likelihood for mu=3 fixed 1.2e−08 1e−08 8e−09 L ( µ ) 6e−09 4e−09 2e−09 0 0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2 8 σ 2

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood When both parameters are uknown, L ( y ; θ = ( µ, σ 2 )) the likelihood is Likelihood for both mu and sigma unknown L ( m u , s i sigma^2 g m a ^ 2 ) mu

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Example 1: Exponential distribution Let y 1 , ., , , y n be a random sample from the exponential density f ( y ; θ ) = θ − 1 e − y /θ ( y > 0 , θ > 0 ) . The parameter space is Θ = R + and the sample space is the Cartesian product R n + . The likelihood is � � n n � � − 1 θ − 1 e − y i /θ = θ − n exp L ( θ ) = y i θ i = 1 i = 1

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Example 2: Cauchy distribution The Cauchy distribution centered at θ is 1 f ( y ; θ ) = π [ 1 + ( y − θ ) 2 ] where y ∈ R and θ ∈ R . The likelihood for a random sample y 1 , ..., y n is � n 1 L ( θ ) = π [ 1 + ( y i − θ ) 2 ] i = 1 The sample space is R n and the parameter space is R .

Likelihood ML estimator Summaries ML properties LR test Profile likelihood Likelihood Example 3: Binomial experiment Consider a random variable R ∼ Binom ( m , π ) m ! r !( m − r )! π r ( 1 − π ) ( m − r ) Pr ( R = r ) = Suppose π depends on a variable x 1 through the relation exp ( β 0 + β 1 x 1 ) π = 1 + exp ( β 0 + β 1 x 1 ) .