Biostatistics 602 - Statistical Inference March 14th, 2013 - PowerPoint PPT Presentation

. .. .. . . .. . . . . . .. . . .. . . . .. .. Biostatistics 602 - Statistical Inference March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang March 14th, 2013 Hyun Min Kang Evaluation of Bayes Estimator Lecture 16 . . . Summary . Consistency Bayes Risk Recap . . . .. . . .. .. . . .. . . . .. . .. . . .. . . . . .. .. . . .. . . .. . . . . . .. . . .. . 1 / 28 . . . . . . . . . . . . . . . . . . . . . . .

• Is a Bayes Estimator the best unbiased estimator? • Compared to other estimators, what are advantages of Bayes • What is conjugate family? • What are the conjugate families of Binomial, Poisson, and Normal . . .. . . .. . . .. .. . . .. . . . .. . Summary March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang distribution? Estimator? Last Lecture . . Consistency Bayes Risk Recap . . . . . .. .. . .. .. . . .. . . . .. . .. . . .. . . . . .. .. . . .. . . .. . . 2 / 28 . . . .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . • What is a Bayes Estimator?

• Compared to other estimators, what are advantages of Bayes • What is conjugate family? • What are the conjugate families of Binomial, Poisson, and Normal . . .. . . .. . . .. .. . . .. . . . .. . Summary March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang distribution? Estimator? Last Lecture . . Consistency Bayes Risk Recap . . . . . .. .. . .. .. . . .. . . . . . .. . . .. . . .. . .. . . . .. . . .. . . .. . . .. . . .. . 2 / 28 . . . . . . . . . . . . . . . . . . . . . . . • What is a Bayes Estimator? • Is a Bayes Estimator the best unbiased estimator?

• What is conjugate family? • What are the conjugate families of Binomial, Poisson, and Normal . . . .. . . .. . . .. .. . .. . . .. . . . Summary March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang distribution? Estimator? Last Lecture . . Consistency Bayes Risk Recap . . . . .. . .. .. .. . . . .. . . . . . .. . . .. . . . .. .. . . .. . . .. . . .. . . .. . . .. . 2 / 28 . . . . . . . . . . . . . . . . . . . . . . . • What is a Bayes Estimator? • Is a Bayes Estimator the best unbiased estimator? • Compared to other estimators, what are advantages of Bayes

• What are the conjugate families of Binomial, Poisson, and Normal . . . .. . . .. . .. .. . . .. . . .. . . . Summary March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang distribution? Estimator? Last Lecture . . Consistency Bayes Risk Recap . . . . .. . .. .. .. . . . .. . . . . . .. . . .. . . . .. .. .. . .. . . .. . . . . . .. . . .. . 2 / 28 . . . . . . . . . . . . . . . . . . . . . . . • What is a Bayes Estimator? • Is a Bayes Estimator the best unbiased estimator? • Compared to other estimators, what are advantages of Bayes • What is conjugate family?

. .. .. . . .. . . . . . .. . . .. . . .. .. Summary March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang distribution? Estimator? Last Lecture . . Consistency Bayes Risk Recap . . . . .. . . .. . . . . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. .. . .. . . .. . . 2 / 28 . . . . . . . . . . . . . . . . . . . . . . . • What is a Bayes Estimator? • Is a Bayes Estimator the best unbiased estimator? • Compared to other estimators, what are advantages of Bayes • What is conjugate family? • What are the conjugate families of Binomial, Poisson, and Normal

. .. .. . . .. . . . . . .. . . .. . . .. .. Summary March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang (Bayes’ rule) Joint Recap - Bayes Estimator . . Consistency Bayes Risk Recap . . . . .. . . .. . . . . .. . . . . . .. . . .. . . .. .. . . .. . . .. . .. . .. 3 / 28 . . .. . . . .. . . . . . . . . . . . . . . . . . . . . . . . • θ : parameter • π ( θ ) : prior distribution • x | θ ∼ f X ( x | θ ) : sampling distribution • Posterior distribution of θ | x Marginal = f X ( x | θ ) π ( θ ) π ( θ | x ) = m ( x ) ∫ m ( x ) = f ( x | θ ) π ( θ ) d θ • Bayes Estimator of θ is ∫ E ( θ | x ) = θπ ( θ | x ) d θ θ ∈ Ω

. .. . .. . . .. . .. .. . . .. . . .. . . . Recap - Example March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang n n i.i.d. Summary . . Consistency Bayes Risk Recap . . . . .. . . .. .. .. . . .. . . . . . .. . . .. . . . . .. .. . .. . . .. . . . 4 / 28 . . . .. .. . . . . . . . . . . . . . . . . . . . . . . . . . • X 1 , · · · , X n ∼ Bernoulli ( p ) • π ( p ) ∼ Beta ( α, β ) α • Prior guess : ˆ p = α + β . • Posterior distribution : π ( p | x ) ∼ Beta ( ∑ x i + α, n − ∑ x i + β ) • Bayes estimator p = α + ∑ x i ∑ x i α + β α ˆ α + β + n = α + β + n + α + β α + β + n

• If • If . .. .. . . .. . . . . . .. . . .. .. . . .. Let March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang , it makes a mistake and loss is not 0. , it makes a correct decision and loss is 0 is an estimator. The mean squared error (MSE) is defined as .. Loss Function Optimality Summary . Consistency Bayes Risk Recap . . . . . . .. . .. . . .. . . . . . .. . . .. . . . 5 / 28 .. .. . . . .. . . . . . . .. . .. . .. . .. . . . . . . . . . . . . . . . . . . . . . . . MSE (ˆ E [ˆ θ − θ ] 2 θ ) =

. .. .. . . .. . . . . . .. .. . .. . . .. .. . March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang The mean squared error (MSE) is defined as Loss Function Optimality Summary Consistency . Bayes Risk Recap . . . . .. . . . . . . . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. .. 5 / 28 . .. . . . . .. . . . . . . . . . . . . . . . . . . . . . . . MSE (ˆ E [ˆ θ − θ ] 2 θ ) = Let ˆ θ is an estimator. • If ˆ θ = θ , it makes a correct decision and loss is 0 • If ˆ θ ̸ = θ , it makes a mistake and loss is not 0.

• Absolute error loss • A loss that penalties overestimation more than underestimation . .. . . .. . . .. . . .. .. . . .. . . . .. L March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang I I L MSE .. Loss Function Summary . Consistency Bayes Risk Recap . . . . . . .. . .. . . .. . . . .. . .. . . .. . . . . . . .. . . .. . . .. . . .. 6 / 28 . .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . Let L ( θ, ˆ θ ) be a function of θ and ˆ θ . • Squared error loss L (ˆ (ˆ θ − θ ) 2 θ, θ ) = Average Loss = E [ L ( θ, ˆ = θ )] which is the expectation of the loss if ˆ θ is used to estimate θ .

• A loss that penalties overestimation more than underestimation . . . . .. . . .. . . .. . . .. . . .. .. . MSE March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang I I L Loss Function . Summary . Consistency Bayes Risk Recap . . . . .. .. . .. .. . . .. . . . . . .. . . .. . . . .. .. . . . .. . . .. . . 6 / 28 .. .. . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . Let L ( θ, ˆ θ ) be a function of θ and ˆ θ . • Squared error loss L (ˆ (ˆ θ − θ ) 2 θ, θ ) = Average Loss = E [ L ( θ, ˆ = θ )] which is the expectation of the loss if ˆ θ is used to estimate θ . • Absolute error loss L (ˆ | ˆ θ ) = θ − θ |

. .. .. .. . .. . . . . . .. . . .. . . .. .. . March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang MSE Loss Function Summary Consistency . Bayes Risk Recap . . . . .. . . . . . . . .. . . .. . . .. . . . .. . . .. 6 / 28 . . . . .. . . .. . .. .. .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . . Let L ( θ, ˆ θ ) be a function of θ and ˆ θ . • Squared error loss L (ˆ (ˆ θ − θ ) 2 θ, θ ) = Average Loss = E [ L ( θ, ˆ = θ )] which is the expectation of the loss if ˆ θ is used to estimate θ . • Absolute error loss L (ˆ | ˆ θ ) = θ − θ | • A loss that penalties overestimation more than underestimation L ( θ, ˆ (ˆ θ − θ ) 2 I (ˆ θ < θ ) + 10(ˆ θ − θ ) 2 I (ˆ θ ) = θ ≥ θ )

. . An estimator with smaller R Risk Function - Average Loss Summary . Consistency Bayes Risk Recap . . . .. preferred. . . .. . . .. .. . .. is . . R March 14th, 2013 Biostatistics 602 - Lecture 16 Hyun Min Kang Bayes risk. respect to a Bayes risk, which is defined as the one that minimize the is the optimal estimator with The Bayes rule with respect to a prior d given prior Definition : Bayes Risk Bayes risk is defined as the average risk across all values of . . . . . . . . . . .. . . . .. . . .. . . .. . .. . . . .. . . .. . . .. . . .. 7 / 28 .. . . . .. . . .. . . . .. . .. . . . .. . . . . . . . . . . . . . . . . . . . . . . . . R ( θ, ˆ E [ L ( θ, ˆ θ ) = θ ( X )) | θ ] If L ( θ, ˆ θ ) = (ˆ θ − θ ) 2 , R ( θ, ˆ θ ) is MSE.