Bayes Statistics: the continuous case Hans-Peter Helfrich - PowerPoint PPT Presentation

Bayes’ Statistics: the continuous case Hans-Peter Helfrich University of Bonn Theodor-Brinkmann-Graduate School H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 1 / 19

Overview Empirical distributions 1 Marginal and conditional density 2 Gaussian prior and likelihood density functions 3 Updating the posterior 4 H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 2 / 19

Multi-dimensional data Data given by [Johnson, 1996] NR FAT BMI ABDOMEN HIP THIGH 1 12.6 23.7 85.2 94.5 59 2 6.9 23.4 83 98.7 58.7 3 24.6 24.7 87.9 99.2 59.6 4 10.9 24.9 86.4 101.2 60.1 5 27.8 25.6 100 101.9 63.2 6 20.6 26.5 94.4 107.8 66 7 19 26.2 90.7 100.3 58.4 8 12.8 23.6 88.5 97.1 60 ... ... Random variables We consider each row as a sample of random variables X 1 , X 2 , . . . , X 5 . In the cited paper N = 252 samples are given. H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 3 / 19

Discrete example Counts Fat / circ. [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 18 18 0 0 0 (10,20] 12 81 6 0 0 (20,30] 0 65 30 0 0 (30,40] 0 2 17 1 1 (40,50] 0 0 0 1 0 Probabilities Fat / circ. [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.071 0.071 0.000 0.000 0.000 (10,20] 0.048 0.321 0.024 0.000 0.000 (20,30] 0.000 0.258 0.119 0.000 0.000 (30,40] 0.000 0.008 0.067 0.004 0.004 (40,50] 0.000 0.000 0.000 0.004 0.000 H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 4 / 19

Discrete example Marginal probabilities Fat / circ. [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.071 0.071 0.000 0.000 0.000 0.143 (10,20] 0.048 0.321 0.024 0.000 0.000 0.393 (20,30] 0.000 0.258 0.119 0.000 0.000 0.377 (30,40] 0.000 0.008 0.067 0.004 0.004 0.083 (40,50] 0.000 0.000 0.000 0.004 0.000 0.004 0.119 0.659 0.210 0.008 0.004 1 Conditional probabilities P ( A , B ) P ( B ∣ A ) = P ( A , B ) P ( A ∣ B ) = P ( B ) , P ( A ) ∑ P ( B ) = P ( B ∣ A j ) P ( A j ) H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 5 / 19

Discrete example Conditional probabilities for fixed fat percentage 휃 [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.500 0.500 0.000 0.000 0.000 0.143 (10,20] 0.121 0.818 0.061 0.000 0.000 0.393 (20,30] 0.000 0.684 0.316 0.000 0.000 0.377 (30,40] 0.000 0.095 0.810 0.048 0.048 0.083 (40,50] 0.000 0.000 0.000 1.000 0.000 0.004 0.119 0.659 0.210 0.008 0.004 1 P (80 < x ≤ 100 ∣ 20 < 휃 ≤ 30) P (20 < 휃 ≤ 30) P (20 < 휃 ≤ 30 ∣ 80 < x ≤ 100) = P (80 < x ≤ 100) 0 . 684 ⋅ 0 . 377 = 0 . 659 = 0 . 392 H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 6 / 19

Discrete example Conditional probabilities for fixed abdomen circumference x [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.600 0.108 0.000 0.000 0.000 (10,20] 0.400 0.488 0.113 0.000 0.000 (20,30] 0.000 0.392 0.566 0.000 0.000 (30,40] 0.000 0.012 0.321 0.500 1.000 (40,50] 0.000 0.000 0.000 0.500 0.000 0.119 0.659 0.210 0.008 0.004 1 H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 7 / 19

Two-dimensional distribution Joint distribution For two random variables, X and Θ, we can assign a joint density 0.5 distribution f ( x , 휃 ). 0.4 D e 0.3 n s i t y [ % 0.2 ] Explanation 0.1 0.0 40 f ( x , 휃 )Δ x Δ 휃 is the probability 140 30 F 120 a t p 20 e c . r c 100 i r e c n n that a value lies in the range t m e a 10 o g d e 80 b A 0 60 ( x , x + Δ x ) , ( 휃, 휃 + Δ 휃 ) Example Figure: Empirical distribution X Abdomen circumference 휃 Fat percentage H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 8 / 19

Empirical distribution f ( x , 휃 ) 40 0.5 0.5 0.4 30 0.4 Density [%] 0.3 Fat percentage 0.3 0.2 20 0.1 0.2 0.0 40 10 140 30 0.1 F 120 a t p 20 e . c r c 100 i r c e n n e t m a 10 g o 80 d e b A 0 0.0 0 60 60 80 100 120 140 Abdomen circ. Figure: Joint distribution of body fat and abdomen c. Data given here H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 9 / 19

Twodimensional normal density distribution 40 0.5 0.5 0.4 30 0.4 Density [%] 0.3 Fat percentage 0.3 20 0.2 0.1 0.2 0.0 40 10 140 30 0.1 F 120 a t p 20 e c . r 100 r c c i e n n e t a m 10 o g d e 80 b A 0 0.0 0 60 60 80 100 120 140 Abdomen circ. Figure: Joint distribution of body fat and abdomen c. H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 10 / 19

Marginal density distribution The marginal distribution gives the distribution of one variable regardless of the other variable. We obtain it by integrating the densities with respect to the other variable. Marginal density distribution for 휃 ∫ p ( 휃 ) = f ( x , 휃 ) dx Marginal densities distribution for x ∫ g ( x ) = f ( x , 휃 ) d 휃 Example p ( 휃 ) Distribution of fat percentage 휃 regardless of abdomen circumference x g ( x ) Distribution of abdomen circumference x H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 11 / 19

Conditional densities Density distribution p ( 휃 ∣ x ) of 휃 for given x p ( 휃 ∣ x ) = f ( x , 휃 ) g ( x ) Density distribution f ( x ∣ 휃 ) of 휃 for given x f ( x ∣ 휃 ) = f ( x , 휃 ) p ( 휃 ) By elimination of f ( x , 휃 ) we come to Bayes’ theorem f ( x ∣ 휃 ) p ( 휃 ) p ( 휃 ∣ x ) = ∫ f ( x ∣ 휃 ) p ( 휃 ) d 휃 H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 12 / 19

Bayes’ theorem Determination of an unknown parameter 휃 Given a prior knowledge by analyzing a former experiment We want to determine an unknown parameter We get further information by a new experiment The outcome should give us a credible interval for the parameter Assumptions The prior distribution p ( 휃 ) of the unknown parameter 휃 An observation x for the unknown parameter The likelihood L ( x ∣ 휃 ) for getting x for a given 휃 Bayes’ theorem L ( x ∣ 휃 ) p ( 휃 ) p ( 휃 ∣ x ) = L ( x ∣ 휃 ) p ( 휃 ) d 휃 ∝ L ( x ∣ 휃 ) p ( 휃 ) ∫ H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 13 / 19

Likelihood and density function Prior density and likelihood We consider the case that the prior and the likelihood are normally distributed cf. [Gelman et al., 2004, Carlin and Louis, 1996] Prior with mean 휇 0 and standard deviation 휎 0 − ( 휃 − 휇 0 ) 2 ( ) 1 = N( 휃 ∣ 휇 0 , 휎 2 p ( 휃 ) = √ exp 0 ) 2 휎 2 2 휋휎 0 0 Likelihood with mean 휃 and standard deviation 휎 − ( x − 휃 ) 2 1 ( ) = N( x ∣ 휃, 휎 2 ) L ( x ∣ 휃 ) = √ exp 2 휎 2 2 휋휎 Later, we will treat more general cases by Monte Carlo simulations. H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 14 / 19

Precision 1 If X is a random variable with standard deviation 휎 , we call 휏 = 휎 2 precision of X . Precision is the reciprocal value of the variance. High precision implies low variance. Theorem Assume p ( 휃 ) has normally distributed density with mean 휇 0 and precision 휏 0 L ( x ∣ 휃 ) has normal distributed density with mean 휇 and precision 휏 Then the posterior density p ( 휃 ∣ x ) is also normal with mean value ¯ 휇 and precision ¯ 휏 휏 0 휏 휇 ¯ = 휏 0 + 휏 휇 0 + 휏 0 + 휏 x 휏 ¯ = 휏 0 + 휏 The mean value ¯ 휇 is the weighted mean of the mean value 휇 0 and the outcome x , the weights are proportional to the precisions. H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 15 / 19

Proof of the theorem We write − ( x − 휃 ) 2 − ( 휃 − 휇 0 ) 2 ( ) ( ) L ( x ∣ 휃 ) ∝ exp , p ( 휃 ) ∝ exp . 2 휎 2 2 휎 2 0 That means the terms on the right and left hand side are proportional, the proportionality constants are independent of x and of 휃 , but may depend on 휎 and 휎 0 . By multiplying both terms, we get − 휏 ( x − 휃 ) 2 − 휏 0 ( 휃 − 휇 0 ) 2 ( ) ( ) p ( 휃 ∣ x ) ∝ L ( x ∣ 휃 ) p ( 휃 ) ∝ exp exp 2 2 ( ) − 1 2( 휏 + 휏 0 )( 휃 2 − 2 휏 x + 휏 0 휇 0 ∝ exp 휃 ) . 휏 + 휏 0 By quadratic completion, we get a normal density function, and can get the mean value and standard deviation by comparing the coefficients. H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 16 / 19

Updating the posterior Several studies For the first study with outcome x 1 , we get by Bayes’ theorem p 1 ( 휃 ∣ x 1 ) ∝ L 1 ( x 1 ∣ 휃 ) p ( 휃 ) Now for the second study, we take p 1 ( 휃 ∣ x 1 ) as a priori density and we obtain p 2 ( 휃 ∣ x 2 ) ∝ L 2 ( x 2 ∣ 휃 ) p 1 ( 휃 ∣ x 1 ) By repeating that procedure for normally distributed priors and likelihoods, we eventually get that p n ( 휃 ∣ x n ) is normally distributed with mean value 휇 = w 0 휇 0 + w 1 x 1 + ⋅ ⋅ ⋅ + w n x n , and precision 휏 = 휏 0 + 휏 1 + ⋅ ⋅ ⋅ + 휏 n , and weights w i = 휏 i /휏, i = 0 , 1 , 2 , ⋅ ⋅ ⋅ , n , where H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 17 / 19