Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Archie Karas • Started playing Razz, like poker but weakest hand wins • Figured it was great for him given his bad luck • Borrowed $10,000 from old friend... • Repaid loan with 50% interest in 3 hours and had plenty left over to play • Kept on playing for 3 years, and seemed unable to lose • Accumulated $40 million in the process (on many games) • Then came the losses: • $11 million playing craps; in 3 weeks • $17 million trying to get back the 11; in 1 week • $2 million playing poker • Caught cheating at a blackjack table in 2013 (4 years of probation) • Placed in Nevada’s Black Book in 2015 Centre for Central Banking Studies Modelling and Forecasting 6
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
Thought experiment • Imagine you are head of security in a casino (pre-2015) • Archie comes through the door... • He wants to play high-stakes coin toss • Every game costs $1 million • Heads: he wins $2 million (stake + $1 million extra) • Tails: he loses his $1 million stake • Archie insists on using his own ‘lucky’ coin • You have limited time and can inspect the coin • How do you decide whether to allow him to play or not? Centre for Central Banking Studies Modelling and Forecasting 7
A primer on maximum likelihood • We really care about the probability of ending up with heads: q ∗ • The probability q ∗ is unknown - Archie may have done something to the coin • Maximum likelihood provides a way of estimating q ∗ Centre for Central Banking Studies Modelling and Forecasting 8
A primer on maximum likelihood • We really care about the probability of ending up with heads: q ∗ • The probability q ∗ is unknown - Archie may have done something to the coin • Maximum likelihood provides a way of estimating q ∗ Centre for Central Banking Studies Modelling and Forecasting 8
A primer on maximum likelihood • We really care about the probability of ending up with heads: q ∗ • The probability q ∗ is unknown - Archie may have done something to the coin • Maximum likelihood provides a way of estimating q ∗ Centre for Central Banking Studies Modelling and Forecasting 8
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Idea • Generate data x by tossing the coin and noting results • We then form a likelihood function L ( q | x ) n � L ( q | x ) ≡ p ( x | q ) = p ( x i | q ) i = 1 • Here • q is number between 0 and 1 • p ( x | q ) denotes the probability of observing x if the probability of heads equals q • p ( x i | q ) is the probabilityof observing x i in throw i conditional on q • A maximum likelihood (ML) estimator of the unknown parameter q ∗ is the argument ˆ q which maximises the likelihood function L ( ·| x ) • Of course ˆ q is conditional on the entire sample x ! Centre for Central Banking Studies Modelling and Forecasting 9
Binomial example • What is the probability of observing tails ( T ) if the probability of heads ( H ) is p ( H ) = q ∗ ? • Single unknown parameter q ∗ to be estimated • Assume we observed a sequence: HTHH . . . TT • What is the corresponding probability as a function of q ∗ ? p ( H | q ∗ ) · p ( T | q ∗ ) · p ( H | q ∗ ) · p ( H | q ∗ ) . . . p ( T | q ∗ ) · p ( T | q ∗ ) Centre for Central Banking Studies Modelling and Forecasting 10
Binomial example • What is the probability of observing tails ( T ) if the probability of heads ( H ) is p ( H ) = q ∗ ? • Single unknown parameter q ∗ to be estimated • Assume we observed a sequence: HTHH . . . TT • What is the corresponding probability as a function of q ∗ ? p ( H | q ∗ ) · p ( T | q ∗ ) · p ( H | q ∗ ) · p ( H | q ∗ ) . . . p ( T | q ∗ ) · p ( T | q ∗ ) Centre for Central Banking Studies Modelling and Forecasting 10
Binomial example • What is the probability of observing tails ( T ) if the probability of heads ( H ) is p ( H ) = q ∗ ? • Single unknown parameter q ∗ to be estimated • Assume we observed a sequence: HTHH . . . TT • What is the corresponding probability as a function of q ∗ ? p ( H | q ∗ ) · p ( T | q ∗ ) · p ( H | q ∗ ) · p ( H | q ∗ ) . . . p ( T | q ∗ ) · p ( T | q ∗ ) Centre for Central Banking Studies Modelling and Forecasting 10
Binomial example • What is the probability of observing tails ( T ) if the probability of heads ( H ) is p ( H ) = q ∗ ? • Single unknown parameter q ∗ to be estimated • Assume we observed a sequence: HTHH . . . TT • What is the corresponding probability as a function of q ∗ ? p ( H | q ∗ ) · p ( T | q ∗ ) · p ( H | q ∗ ) · p ( H | q ∗ ) . . . p ( T | q ∗ ) · p ( T | q ∗ ) Centre for Central Banking Studies Modelling and Forecasting 10
Binomial example (ctd.) • To estimate q ∗ define the likelihood L ( q | HTHH . . . TT ) = p ( H | q ) · p ( T | q ) · p ( H | q ) · p ( H | q ) . . . . . . p ( T | q ) · p ( T | q ) = q k ( 1 − q ) n − k • What are k and n ? • The ML estimator of the unknown parameter q ∗ is the ˆ q which maximises the likelihood function L ( q | HTHH . . . TT ) Centre for Central Banking Studies Modelling and Forecasting 11
Binomial example (ctd.) • To estimate q ∗ define the likelihood L ( q | HTHH . . . TT ) = p ( H | q ) · p ( T | q ) · p ( H | q ) · p ( H | q ) . . . . . . p ( T | q ) · p ( T | q ) = q k ( 1 − q ) n − k • What are k and n ? • The ML estimator of the unknown parameter q ∗ is the ˆ q which maximises the likelihood function L ( q | HTHH . . . TT ) Centre for Central Banking Studies Modelling and Forecasting 11
Binomial example (ctd.) • To estimate q ∗ define the likelihood L ( q | HTHH . . . TT ) = p ( H | q ) · p ( T | q ) · p ( H | q ) · p ( H | q ) . . . . . . p ( T | q ) · p ( T | q ) = q k ( 1 − q ) n − k • What are k and n ? • The ML estimator of the unknown parameter q ∗ is the ˆ q which maximises the likelihood function L ( q | HTHH . . . TT ) Centre for Central Banking Studies Modelling and Forecasting 11
Binomial example (ctd.) • Exercise 1. Find the ML estimator of q ∗ conditional on observing the sequence H , H , H , T , T , H , T , H , T , H • What is the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ? • We can plot this function to see if it has a maximum • Important to understand what this function shows us! Centre for Central Banking Studies Modelling and Forecasting 12
Binomial example (ctd.) • Exercise 1. Find the ML estimator of q ∗ conditional on observing the sequence H , H , H , T , T , H , T , H , T , H • What is the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ? • We can plot this function to see if it has a maximum • Important to understand what this function shows us! Centre for Central Banking Studies Modelling and Forecasting 12
Binomial example (ctd.) • Exercise 1. Find the ML estimator of q ∗ conditional on observing the sequence H , H , H , T , T , H , T , H , T , H • What is the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ? • We can plot this function to see if it has a maximum • Important to understand what this function shows us! Centre for Central Banking Studies Modelling and Forecasting 12
Binomial example (ctd.) • Exercise 1. Find the ML estimator of q ∗ conditional on observing the sequence H , H , H , T , T , H , T , H , T , H • What is the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ? • We can plot this function to see if it has a maximum • Important to understand what this function shows us! Centre for Central Banking Studies Modelling and Forecasting 12
Binomial example (ctd.) −3 1.2 x 10 1 0.8 0.6 0.4 0.2 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Centre for Central Banking Studies Modelling and Forecasting 13
Binomial example (ctd.) • We could solve for the maximum analytically • To do so take the derivative of the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ q 6 ( 1 − q ) 4 with respect to q and set it equal to zero ∂ L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ 6 q 5 ( 1 − q ) 4 − 4 q 6 ( 1 − q ) 3 ∂ q = q 5 ( 1 − q ) 3 ( 6 ( 1 − q ) − 4 q ) ≡ 0 • Eliminating 0 and 1 as maximum candidates (why?) = ⇒ 6 ( 1 − ˆ q ) − 4 ˆ q = 0 ⇔ 10 ˆ q = 6 ⇔ ˆ q = 0 . 6 • ML estimator of q ∗ is 0.6! • Note that this does not account for any prior beliefs we may have had about Archie being dishonest Centre for Central Banking Studies Modelling and Forecasting 14
Binomial example (ctd.) • We could solve for the maximum analytically • To do so take the derivative of the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ q 6 ( 1 − q ) 4 with respect to q and set it equal to zero ∂ L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ 6 q 5 ( 1 − q ) 4 − 4 q 6 ( 1 − q ) 3 ∂ q = q 5 ( 1 − q ) 3 ( 6 ( 1 − q ) − 4 q ) ≡ 0 • Eliminating 0 and 1 as maximum candidates (why?) = ⇒ 6 ( 1 − ˆ q ) − 4 ˆ q = 0 ⇔ 10 ˆ q = 6 ⇔ ˆ q = 0 . 6 • ML estimator of q ∗ is 0.6! • Note that this does not account for any prior beliefs we may have had about Archie being dishonest Centre for Central Banking Studies Modelling and Forecasting 14
Binomial example (ctd.) • We could solve for the maximum analytically • To do so take the derivative of the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ q 6 ( 1 − q ) 4 with respect to q and set it equal to zero ∂ L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ 6 q 5 ( 1 − q ) 4 − 4 q 6 ( 1 − q ) 3 ∂ q = q 5 ( 1 − q ) 3 ( 6 ( 1 − q ) − 4 q ) ≡ 0 • Eliminating 0 and 1 as maximum candidates (why?) = ⇒ 6 ( 1 − ˆ q ) − 4 ˆ q = 0 ⇔ 10 ˆ q = 6 ⇔ ˆ q = 0 . 6 • ML estimator of q ∗ is 0.6! • Note that this does not account for any prior beliefs we may have had about Archie being dishonest Centre for Central Banking Studies Modelling and Forecasting 14
Binomial example (ctd.) • We could solve for the maximum analytically • To do so take the derivative of the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ q 6 ( 1 − q ) 4 with respect to q and set it equal to zero ∂ L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ 6 q 5 ( 1 − q ) 4 − 4 q 6 ( 1 − q ) 3 ∂ q = q 5 ( 1 − q ) 3 ( 6 ( 1 − q ) − 4 q ) ≡ 0 • Eliminating 0 and 1 as maximum candidates (why?) = ⇒ 6 ( 1 − ˆ q ) − 4 ˆ q = 0 ⇔ 10 ˆ q = 6 ⇔ ˆ q = 0 . 6 • ML estimator of q ∗ is 0.6! • Note that this does not account for any prior beliefs we may have had about Archie being dishonest Centre for Central Banking Studies Modelling and Forecasting 14
Binomial example (ctd.) • We could solve for the maximum analytically • To do so take the derivative of the likelihood function L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ q 6 ( 1 − q ) 4 with respect to q and set it equal to zero ∂ L ( q | H , H , H , T , T , H , T , H , T , H ) ≡ 6 q 5 ( 1 − q ) 4 − 4 q 6 ( 1 − q ) 3 ∂ q = q 5 ( 1 − q ) 3 ( 6 ( 1 − q ) − 4 q ) ≡ 0 • Eliminating 0 and 1 as maximum candidates (why?) = ⇒ 6 ( 1 − ˆ q ) − 4 ˆ q = 0 ⇔ 10 ˆ q = 6 ⇔ ˆ q = 0 . 6 • ML estimator of q ∗ is 0.6! • Note that this does not account for any prior beliefs we may have had about Archie being dishonest Centre for Central Banking Studies Modelling and Forecasting 14
Univariate regression • Now, consider a univariate regression model, where B ∗ and σ ∗ are to be estimated y t = B ∗ x t + v t , v t ∼ N . i . d . ( 0 , ( σ ∗ ) 2 ) Independence and the assumption of normality imply � � 1 1 2 ( σ ∗ ) 2 v 2 p ( v t ) = √ 2 πσ ∗ exp − t • What is the likelihood function L ( B , σ 2 | x 1 , x 2 , . . . , x n , y 1 , y 2 , . . . , y n ) ? • Idea: estimate B ∗ and σ ∗ by finding the maximum of the likelihood function with respect to B and σ • Just like the coin toss example except now we have two parameters! • In what follows x ≡ ( x 1 , x 2 , . . . , x n ) , y ≡ ( y 1 , y 2 , . . . , y n ) Centre for Central Banking Studies Modelling and Forecasting 15
Univariate regression • Now, consider a univariate regression model, where B ∗ and σ ∗ are to be estimated y t = B ∗ x t + v t , v t ∼ N . i . d . ( 0 , ( σ ∗ ) 2 ) Independence and the assumption of normality imply � � 1 1 2 ( σ ∗ ) 2 v 2 p ( v t ) = √ 2 πσ ∗ exp − t • What is the likelihood function L ( B , σ 2 | x 1 , x 2 , . . . , x n , y 1 , y 2 , . . . , y n ) ? • Idea: estimate B ∗ and σ ∗ by finding the maximum of the likelihood function with respect to B and σ • Just like the coin toss example except now we have two parameters! • In what follows x ≡ ( x 1 , x 2 , . . . , x n ) , y ≡ ( y 1 , y 2 , . . . , y n ) Centre for Central Banking Studies Modelling and Forecasting 15
Univariate regression • Now, consider a univariate regression model, where B ∗ and σ ∗ are to be estimated y t = B ∗ x t + v t , v t ∼ N . i . d . ( 0 , ( σ ∗ ) 2 ) Independence and the assumption of normality imply � � 1 1 2 ( σ ∗ ) 2 v 2 p ( v t ) = √ 2 πσ ∗ exp − t • What is the likelihood function L ( B , σ 2 | x 1 , x 2 , . . . , x n , y 1 , y 2 , . . . , y n ) ? • Idea: estimate B ∗ and σ ∗ by finding the maximum of the likelihood function with respect to B and σ • Just like the coin toss example except now we have two parameters! • In what follows x ≡ ( x 1 , x 2 , . . . , x n ) , y ≡ ( y 1 , y 2 , . . . , y n ) Centre for Central Banking Studies Modelling and Forecasting 15
Univariate regression • Now, consider a univariate regression model, where B ∗ and σ ∗ are to be estimated y t = B ∗ x t + v t , v t ∼ N . i . d . ( 0 , ( σ ∗ ) 2 ) Independence and the assumption of normality imply � � 1 1 2 ( σ ∗ ) 2 v 2 p ( v t ) = √ 2 πσ ∗ exp − t • What is the likelihood function L ( B , σ 2 | x 1 , x 2 , . . . , x n , y 1 , y 2 , . . . , y n ) ? • Idea: estimate B ∗ and σ ∗ by finding the maximum of the likelihood function with respect to B and σ • Just like the coin toss example except now we have two parameters! • In what follows x ≡ ( x 1 , x 2 , . . . , x n ) , y ≡ ( y 1 , y 2 , . . . , y n ) Centre for Central Banking Studies Modelling and Forecasting 15
Univariate regression • Now, consider a univariate regression model, where B ∗ and σ ∗ are to be estimated y t = B ∗ x t + v t , v t ∼ N . i . d . ( 0 , ( σ ∗ ) 2 ) Independence and the assumption of normality imply � � 1 1 2 ( σ ∗ ) 2 v 2 p ( v t ) = √ 2 πσ ∗ exp − t • What is the likelihood function L ( B , σ 2 | x 1 , x 2 , . . . , x n , y 1 , y 2 , . . . , y n ) ? • Idea: estimate B ∗ and σ ∗ by finding the maximum of the likelihood function with respect to B and σ • Just like the coin toss example except now we have two parameters! • In what follows x ≡ ( x 1 , x 2 , . . . , x n ) , y ≡ ( y 1 , y 2 , . . . , y n ) Centre for Central Banking Studies Modelling and Forecasting 15
ML estimates of population parameters (ctd.) • Turns out to be easier to differentiate the log of the likelihood rather than the likelihood itself • This is equivalent, since L ( B , σ 2 | x , y ) > 0 and because � � ∂ 1 ∂ L ( B , σ 2 | x , y ) ∂ B L ( B , σ 2 | x , y ) ∂ B ln = L ( B , σ 2 | x , y ) • How does this help us? • We then have � ( y i − Bx i ) 2 � � � n � = − n 2 ln ( 2 π ) − n 2 ln σ 2 − 1 L ( B , σ 2 | x , y ) ln σ 2 2 i = 1 Centre for Central Banking Studies Modelling and Forecasting 16
ML estimates of population parameters (ctd.) • Turns out to be easier to differentiate the log of the likelihood rather than the likelihood itself • This is equivalent, since L ( B , σ 2 | x , y ) > 0 and because � � ∂ 1 ∂ L ( B , σ 2 | x , y ) ∂ B L ( B , σ 2 | x , y ) ∂ B ln = L ( B , σ 2 | x , y ) • How does this help us? • We then have � ( y i − Bx i ) 2 � � � n � = − n 2 ln ( 2 π ) − n 2 ln σ 2 − 1 L ( B , σ 2 | x , y ) ln σ 2 2 i = 1 Centre for Central Banking Studies Modelling and Forecasting 16
ML estimates of population parameters (ctd.) • Turns out to be easier to differentiate the log of the likelihood rather than the likelihood itself • This is equivalent, since L ( B , σ 2 | x , y ) > 0 and because � � ∂ 1 ∂ L ( B , σ 2 | x , y ) ∂ B L ( B , σ 2 | x , y ) ∂ B ln = L ( B , σ 2 | x , y ) • How does this help us? • We then have � ( y i − Bx i ) 2 � � � n � = − n 2 ln ( 2 π ) − n 2 ln σ 2 − 1 L ( B , σ 2 | x , y ) ln σ 2 2 i = 1 Centre for Central Banking Studies Modelling and Forecasting 16
ML estimates of population parameters (ctd.) • Turns out to be easier to differentiate the log of the likelihood rather than the likelihood itself • This is equivalent, since L ( B , σ 2 | x , y ) > 0 and because � � ∂ 1 ∂ L ( B , σ 2 | x , y ) ∂ B L ( B , σ 2 | x , y ) ∂ B ln = L ( B , σ 2 | x , y ) • How does this help us? • We then have � ( y i − Bx i ) 2 � � � n � = − n 2 ln ( 2 π ) − n 2 ln σ 2 − 1 L ( B , σ 2 | x , y ) ln σ 2 2 i = 1 Centre for Central Banking Studies Modelling and Forecasting 16
ML estimates of population parameters (ctd.) • The first order conditions, or the likelihood equations are � � � � n L ( B , σ 2 | x , y ) � ∂ ln = 1 � ( y i − ˆ Bx i ) x i = 0 � σ 2 ∂ B � ˆ (ˆ i = 1 B , ˆ σ ) � � � � n � L ( B , σ 2 | x , y ) ∂ ln = − n 1 � Bx i ) 2 = 0 ( y i − ˆ � σ 2 + ∂σ 2 σ 4 � 2 ˆ 2 ˆ (ˆ i = 1 B , ˆ σ ) Centre for Central Banking Studies Modelling and Forecasting 17
ML estimates of population parameters (ctd.) • From the first of these � n � − 1 � � n n � � � � � − 1 � � y i x i − ˆ = 0 ⇒ ˆ x ′ x x ′ y Bx i x i B = x i x i x i y i = i = 1 i = 1 i = 1 • The maximum likelihood estimate of B ∗ is therefore ˆ B given by the familiar formula above • From the second we obtain � � ′ � � y − ˆ y − ˆ n n B x B x � � − n + 1 σ 2 = 1 Bx i ) 2 = 0 ⇒ ˆ Bx i ) 2 = ( y i − ˆ ( y i − ˆ σ 2 n n ˆ i = 1 i = 1 • ML estimate of σ 2 divided through by n (not n − k ) so biased in small samples but not asymptotically • This is quite typical of ML estimates Centre for Central Banking Studies Modelling and Forecasting 18
ML estimates of population parameters (ctd.) • From the first of these � n � − 1 � � n n � � � � � − 1 � � y i x i − ˆ = 0 ⇒ ˆ x ′ x x ′ y Bx i x i B = x i x i x i y i = i = 1 i = 1 i = 1 • The maximum likelihood estimate of B ∗ is therefore ˆ B given by the familiar formula above • From the second we obtain � � ′ � � y − ˆ y − ˆ n n B x B x � � − n + 1 σ 2 = 1 Bx i ) 2 = 0 ⇒ ˆ Bx i ) 2 = ( y i − ˆ ( y i − ˆ σ 2 n n ˆ i = 1 i = 1 • ML estimate of σ 2 divided through by n (not n − k ) so biased in small samples but not asymptotically • This is quite typical of ML estimates Centre for Central Banking Studies Modelling and Forecasting 18
ML estimates of population parameters (ctd.) • From the first of these � n � − 1 � � n n � � � � � − 1 � � y i x i − ˆ = 0 ⇒ ˆ x ′ x x ′ y Bx i x i B = x i x i x i y i = i = 1 i = 1 i = 1 • The maximum likelihood estimate of B ∗ is therefore ˆ B given by the familiar formula above • From the second we obtain � � ′ � � y − ˆ y − ˆ n n B x B x � � − n + 1 σ 2 = 1 Bx i ) 2 = 0 ⇒ ˆ Bx i ) 2 = ( y i − ˆ ( y i − ˆ σ 2 n n ˆ i = 1 i = 1 • ML estimate of σ 2 divided through by n (not n − k ) so biased in small samples but not asymptotically • This is quite typical of ML estimates Centre for Central Banking Studies Modelling and Forecasting 18
ML estimates of population parameters (ctd.) • From the first of these � n � − 1 � � n n � � � � � − 1 � � y i x i − ˆ = 0 ⇒ ˆ x ′ x x ′ y Bx i x i B = x i x i x i y i = i = 1 i = 1 i = 1 • The maximum likelihood estimate of B ∗ is therefore ˆ B given by the familiar formula above • From the second we obtain � � ′ � � y − ˆ y − ˆ n n B x B x � � − n + 1 σ 2 = 1 Bx i ) 2 = 0 ⇒ ˆ Bx i ) 2 = ( y i − ˆ ( y i − ˆ σ 2 n n ˆ i = 1 i = 1 • ML estimate of σ 2 divided through by n (not n − k ) so biased in small samples but not asymptotically • This is quite typical of ML estimates Centre for Central Banking Studies Modelling and Forecasting 18
ML estimates of population parameters (ctd.) • From the first of these � n � − 1 � � n n � � � � � − 1 � � y i x i − ˆ = 0 ⇒ ˆ x ′ x x ′ y Bx i x i B = x i x i x i y i = i = 1 i = 1 i = 1 • The maximum likelihood estimate of B ∗ is therefore ˆ B given by the familiar formula above • From the second we obtain � � ′ � � y − ˆ y − ˆ n n B x B x � � − n + 1 σ 2 = 1 Bx i ) 2 = 0 ⇒ ˆ Bx i ) 2 = ( y i − ˆ ( y i − ˆ σ 2 n n ˆ i = 1 i = 1 • ML estimate of σ 2 divided through by n (not n − k ) so biased in small samples but not asymptotically • This is quite typical of ML estimates Centre for Central Banking Studies Modelling and Forecasting 18
Summary of ML approach to linear regression • We wanted to estimate B ∗ and σ ∗ in the following model v t ∼ N ( 0 , ( σ ∗ ) 2 ) y t = B ∗ x t + v t , • To do that, we wrote down the likelihood function � � � 2 πσ 2 � − n / 2 − ( y − B x ) ′ ( y − B x ) L ( B , σ 2 | x , y ) = exp 2 σ 2 • Plugging in data and maximising L ( B , σ 2 | x , y ) with respect to B and σ 2 gave us the standard OLS estimator � � ′ � � B = ( x ′ x ) − 1 ( x ′ y ) and ˆ σ 2 = ˆ y − ˆ y − ˆ B x B x / n • The procedure only incorporated information in x and y ! • The Bayesian approach will allow us to combine prior beliefs about B ∗ and σ ∗ with information in x and y Centre for Central Banking Studies Modelling and Forecasting 19
Summary of ML approach to linear regression • We wanted to estimate B ∗ and σ ∗ in the following model v t ∼ N ( 0 , ( σ ∗ ) 2 ) y t = B ∗ x t + v t , • To do that, we wrote down the likelihood function � � � 2 πσ 2 � − n / 2 − ( y − B x ) ′ ( y − B x ) L ( B , σ 2 | x , y ) = exp 2 σ 2 • Plugging in data and maximising L ( B , σ 2 | x , y ) with respect to B and σ 2 gave us the standard OLS estimator � � ′ � � B = ( x ′ x ) − 1 ( x ′ y ) and ˆ σ 2 = ˆ y − ˆ y − ˆ B x B x / n • The procedure only incorporated information in x and y ! • The Bayesian approach will allow us to combine prior beliefs about B ∗ and σ ∗ with information in x and y Centre for Central Banking Studies Modelling and Forecasting 19
Summary of ML approach to linear regression • We wanted to estimate B ∗ and σ ∗ in the following model v t ∼ N ( 0 , ( σ ∗ ) 2 ) y t = B ∗ x t + v t , • To do that, we wrote down the likelihood function � � � 2 πσ 2 � − n / 2 − ( y − B x ) ′ ( y − B x ) L ( B , σ 2 | x , y ) = exp 2 σ 2 • Plugging in data and maximising L ( B , σ 2 | x , y ) with respect to B and σ 2 gave us the standard OLS estimator � � ′ � � B = ( x ′ x ) − 1 ( x ′ y ) and ˆ σ 2 = ˆ y − ˆ y − ˆ B x B x / n • The procedure only incorporated information in x and y ! • The Bayesian approach will allow us to combine prior beliefs about B ∗ and σ ∗ with information in x and y Centre for Central Banking Studies Modelling and Forecasting 19
Summary of ML approach to linear regression • We wanted to estimate B ∗ and σ ∗ in the following model v t ∼ N ( 0 , ( σ ∗ ) 2 ) y t = B ∗ x t + v t , • To do that, we wrote down the likelihood function � � � 2 πσ 2 � − n / 2 − ( y − B x ) ′ ( y − B x ) L ( B , σ 2 | x , y ) = exp 2 σ 2 • Plugging in data and maximising L ( B , σ 2 | x , y ) with respect to B and σ 2 gave us the standard OLS estimator � � ′ � � B = ( x ′ x ) − 1 ( x ′ y ) and ˆ σ 2 = ˆ y − ˆ y − ˆ B x B x / n • The procedure only incorporated information in x and y ! • The Bayesian approach will allow us to combine prior beliefs about B ∗ and σ ∗ with information in x and y Centre for Central Banking Studies Modelling and Forecasting 19
Summary of ML approach to linear regression • We wanted to estimate B ∗ and σ ∗ in the following model v t ∼ N ( 0 , ( σ ∗ ) 2 ) y t = B ∗ x t + v t , • To do that, we wrote down the likelihood function � � � 2 πσ 2 � − n / 2 − ( y − B x ) ′ ( y − B x ) L ( B , σ 2 | x , y ) = exp 2 σ 2 • Plugging in data and maximising L ( B , σ 2 | x , y ) with respect to B and σ 2 gave us the standard OLS estimator � � ′ � � B = ( x ′ x ) − 1 ( x ′ y ) and ˆ σ 2 = ˆ y − ˆ y − ˆ B x B x / n • The procedure only incorporated information in x and y ! • The Bayesian approach will allow us to combine prior beliefs about B ∗ and σ ∗ with information in x and y Centre for Central Banking Studies Modelling and Forecasting 19
ML score and information • At the optimum there are two matrices we will use later in testing • The efficient score matrix is ∂ ln L ( θ ) = S ( θ ) ∂θ • This should be zero at ML estimate • The information matrix is given by � � − ∂ ln L ( θ ) E = I ( θ ) ∂θ∂θ ′ Centre for Central Banking Studies Modelling and Forecasting 20
ML score and information • At the optimum there are two matrices we will use later in testing • The efficient score matrix is ∂ ln L ( θ ) = S ( θ ) ∂θ • This should be zero at ML estimate • The information matrix is given by � � − ∂ ln L ( θ ) E = I ( θ ) ∂θ∂θ ′ Centre for Central Banking Studies Modelling and Forecasting 20
ML score and information • At the optimum there are two matrices we will use later in testing • The efficient score matrix is ∂ ln L ( θ ) = S ( θ ) ∂θ • This should be zero at ML estimate • The information matrix is given by � � − ∂ ln L ( θ ) E = I ( θ ) ∂θ∂θ ′ Centre for Central Banking Studies Modelling and Forecasting 20
ML score and information • At the optimum there are two matrices we will use later in testing • The efficient score matrix is ∂ ln L ( θ ) = S ( θ ) ∂θ • This should be zero at ML estimate • The information matrix is given by � � − ∂ ln L ( θ ) E = I ( θ ) ∂θ∂θ ′ Centre for Central Banking Studies Modelling and Forecasting 20
ML score and information • I ( θ ) turns out to be the most important matrix, as the Cram´ er-Rao lower bound states that − 1 Var (ˆ θ ml ) ≥ [ I (ˆ θ ml )] • This means that the covariance of an estimate exceeds the inverse of the information matrix by a positive semi-definite matrix • Thus the inverse provides the lower bound that an estimate can achieve! • Fortunately, a large number of ML estimators achieve this lower bound • I.e. in many cases it can be shown that the inverse of the information matrix is the covariance of the estimate er-Rao lower bound gives the covariance of ˆ • In those cases the Cram´ θ Centre for Central Banking Studies Modelling and Forecasting 21
ML score and information • I ( θ ) turns out to be the most important matrix, as the Cram´ er-Rao lower bound states that − 1 Var (ˆ θ ml ) ≥ [ I (ˆ θ ml )] • This means that the covariance of an estimate exceeds the inverse of the information matrix by a positive semi-definite matrix • Thus the inverse provides the lower bound that an estimate can achieve! • Fortunately, a large number of ML estimators achieve this lower bound • I.e. in many cases it can be shown that the inverse of the information matrix is the covariance of the estimate er-Rao lower bound gives the covariance of ˆ • In those cases the Cram´ θ Centre for Central Banking Studies Modelling and Forecasting 21
ML score and information • I ( θ ) turns out to be the most important matrix, as the Cram´ er-Rao lower bound states that − 1 Var (ˆ θ ml ) ≥ [ I (ˆ θ ml )] • This means that the covariance of an estimate exceeds the inverse of the information matrix by a positive semi-definite matrix • Thus the inverse provides the lower bound that an estimate can achieve! • Fortunately, a large number of ML estimators achieve this lower bound • I.e. in many cases it can be shown that the inverse of the information matrix is the covariance of the estimate er-Rao lower bound gives the covariance of ˆ • In those cases the Cram´ θ Centre for Central Banking Studies Modelling and Forecasting 21
ML score and information • I ( θ ) turns out to be the most important matrix, as the Cram´ er-Rao lower bound states that − 1 Var (ˆ θ ml ) ≥ [ I (ˆ θ ml )] • This means that the covariance of an estimate exceeds the inverse of the information matrix by a positive semi-definite matrix • Thus the inverse provides the lower bound that an estimate can achieve! • Fortunately, a large number of ML estimators achieve this lower bound • I.e. in many cases it can be shown that the inverse of the information matrix is the covariance of the estimate er-Rao lower bound gives the covariance of ˆ • In those cases the Cram´ θ Centre for Central Banking Studies Modelling and Forecasting 21
ML score and information • I ( θ ) turns out to be the most important matrix, as the Cram´ er-Rao lower bound states that − 1 Var (ˆ θ ml ) ≥ [ I (ˆ θ ml )] • This means that the covariance of an estimate exceeds the inverse of the information matrix by a positive semi-definite matrix • Thus the inverse provides the lower bound that an estimate can achieve! • Fortunately, a large number of ML estimators achieve this lower bound • I.e. in many cases it can be shown that the inverse of the information matrix is the covariance of the estimate er-Rao lower bound gives the covariance of ˆ • In those cases the Cram´ θ Centre for Central Banking Studies Modelling and Forecasting 21
ML score and information • I ( θ ) turns out to be the most important matrix, as the Cram´ er-Rao lower bound states that − 1 Var (ˆ θ ml ) ≥ [ I (ˆ θ ml )] • This means that the covariance of an estimate exceeds the inverse of the information matrix by a positive semi-definite matrix • Thus the inverse provides the lower bound that an estimate can achieve! • Fortunately, a large number of ML estimators achieve this lower bound • I.e. in many cases it can be shown that the inverse of the information matrix is the covariance of the estimate er-Rao lower bound gives the covariance of ˆ • In those cases the Cram´ θ Centre for Central Banking Studies Modelling and Forecasting 21
Asymptotic inference • We expect that any normalised statistic is distributed as √ θ − θ ) d n ( � → N ( 0 , I A (ˆ θ ) − 1 ) • Where I A ( θ ) − 1 = plim n I ( θ ) − 1 • Usually replace n I A ( θ ) − 1 by I ( θ ) − 1 • If we can formulate any statistic like this we know its distribution is χ 2 m with m the number of restrictions • Typically consider Wald (W), Likelihood ratio (LR) and LM statistics • Quick look at LR here Centre for Central Banking Studies Modelling and Forecasting 22
Asymptotic inference • We expect that any normalised statistic is distributed as √ θ − θ ) d n ( � → N ( 0 , I A (ˆ θ ) − 1 ) • Where I A ( θ ) − 1 = plim n I ( θ ) − 1 • Usually replace n I A ( θ ) − 1 by I ( θ ) − 1 • If we can formulate any statistic like this we know its distribution is χ 2 m with m the number of restrictions • Typically consider Wald (W), Likelihood ratio (LR) and LM statistics • Quick look at LR here Centre for Central Banking Studies Modelling and Forecasting 22
Asymptotic inference • We expect that any normalised statistic is distributed as √ θ − θ ) d n ( � → N ( 0 , I A (ˆ θ ) − 1 ) • Where I A ( θ ) − 1 = plim n I ( θ ) − 1 • Usually replace n I A ( θ ) − 1 by I ( θ ) − 1 • If we can formulate any statistic like this we know its distribution is χ 2 m with m the number of restrictions • Typically consider Wald (W), Likelihood ratio (LR) and LM statistics • Quick look at LR here Centre for Central Banking Studies Modelling and Forecasting 22
Recommend
More recommend
