Motivation Simulation Diagnosis Workfmow Example Testing MCMC Samplers Jason M.T. Roos First European Bayesian Summit in Marketing Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example MCMC is awful ▶ The probability of any computer program having a bug is strictly positive ▶ Your MCMC sampler is a computer program Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example But my code is bug-free!!!!1 1. Lucky you 2. Bug-free MCMC code can fail to sample from the model due to ▶ Non-identifjability ▶ Pathological behavior of the sampling algorithm ▶ Both of which are diffjcult to spot while coding Testing MCMC Samplers Jason M.T. Roos
𝑇 ∶ 𝑧 ↦ 𝜄 1∶𝑀 , 𝑞(𝜄|𝑧) ∝ 𝜌(𝑧|𝜄)𝜌(𝜄), Motivation Simulation Diagnosis Workfmow Example The crux of the problem ▶ We have a model, that we cannot evaluate directly ▶ We have a sampling procedure, 𝜄 1∶𝑀 ≡ {𝜄 1 , … , 𝜄 𝑀 }, implemented by people who are not trained to write software ▶ How do we know that ▶ [STRONG] 𝑇 is implemented correctly, or ▶ [WEAK] 𝑇 is probably not implemented incorrectly; and hence ▶ 𝜄 1∶𝑀 is (approximately) distributed according to 𝑞(𝜄|𝑧) ? Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example Agenda 1. Some theory about testing MCMC samplers 2. A detailed example showing how testing can be a valuable part of your workfmow Testing MCMC Samplers Jason M.T. Roos
𝑧) ̃ ̃ ̃ ̃ Motivation Simulation Diagnosis Workfmow Example Simple approach to testing ▶ Typical (among papers that even mention validating code) ▶ Choose 𝜄 and call it the “true” value ▶ Generate ̃ 𝜄) according to the data-generating process (DGP, 𝑧 ∼ 𝜌(𝑧| ̃ i.e., likelihood) ▶ Run the MCMC sampler to get 𝜄 1∶𝑀 ∶= 𝑇( ̃ ▶ If 𝜄 is typical of the 𝜄 1∶𝑀 ’s, then assume the program 𝑇 is working as expected ▶ What is “typical”? Maybe 𝜄 is within the 95% CI for 𝜄 1∶𝑀 ▶ Because 𝑞(𝜄|𝑧) can include values of 𝜄 with low probability, any individual simulation is not informative ▶ Hence, choose a few of values of 𝜄 and show, on average, they are not different from their corresponding MCMC samples 𝜄 1∶𝑀 Testing MCMC Samplers Jason M.T. Roos
̃ 𝜄) 𝑧) 𝜄 𝑧) ̃ 𝜄) Motivation Simulation Diagnosis Workfmow Example Exploiting the model structure ▶ Bayes models have a nice property that allows a better approach for choosing 𝜄 (Geweke 2004; Cook, Gelman, and Rubin 2006) 1. Draw 𝜄 ∼ 𝜌(𝜄) from its prior distribution 2. Simulate ̃ 𝜄) according to the DGP 𝑧 ∼ 𝜌(𝑧| ̃ 3. Run the sampler 𝑇 to obtain MCMC samples 𝜄 1∶𝑀 ∶= 𝑇( ̃ ▶ The MCMC samples 𝜄 1∶𝑀 are draws from the prior distribution 𝜌(𝜄) because 𝑧| ̃ 𝜌( ̃ 𝑧 𝑒 ̃ 𝜌(𝜄) = 𝜌(𝜄| ̃ 𝜌( ̃ 𝑒 ̃ Step 1 Step 3 Step 2 Testing MCMC Samplers Jason M.T. Roos
ℓ=1 𝜄) = ̃ 𝜄) ∈ [0, 𝑀], ̃ 𝑀 ̃ Motivation Simulation Diagnosis Workfmow Example Comparing MCMC samples to the true prior ▶ Discrepancy between the “true” 𝜄 ’s and their corresponding MCMC samples 𝜄 1∶𝑀 suggests a problem with 𝑇 ▶ Use discrete rank statistics, 𝑠(𝜄 1∶𝑀 , ̃ 𝟚(𝜄 ℓ < to test for such discrepancies Theorem 1 (Talts et al. 2018). Let 𝜄 ∼ 𝜌(𝜄) , ̃ 𝜄) , and {𝜄 1 , … , 𝜄 𝑀 } ∼ 𝜌(𝜄| ̃ 𝑧) for any distribution 𝑧 ∼ 𝜌(𝑧| ̃ 𝜌(𝑧, 𝜄) . The rank statistics of any one-dimensional random variable over 𝜄 is uniformly distributed over the integers [0, 𝑀] . Testing MCMC Samplers Jason M.T. Roos
𝑧) ̃ Motivation Simulation Diagnosis Workfmow Example Testing procedure for a one-dimensional parameter 𝜄 ▶ Initialize a histogram 𝐼 with 𝑀 + 1 bins ▶ Conduct 𝑂 replications of the following: 1. Draw 𝜄 ∼ 𝜌(𝜄) from its prior distribution 2. Simulate ̃ 𝜄) according to the DGP 𝑧 ∼ 𝜌(𝑧| ̃ 3. Run the sampler 𝑇 to obtain MCMC samples 𝜄 1∶𝑀 ∶= 𝑇( ̃ 4. Set 𝑠 ∶= 𝑠(𝜄 1∶𝑀 , ̃ 𝜄) and increment the bin 𝐼 𝑠 by 1 ▶ The bin 𝐼 0 counts the number of times 𝑠 = 0 , 𝐼 1 counts the number of times 𝑠 = 1 , etc. ▶ The counts in the 𝑀 + 1 bins are distributed approximately 𝐼 𝑠 ∼ 𝐶𝑗𝑜(𝑂, (𝑀 + 1) −1 ) Testing MCMC Samplers Jason M.T. Roos
̃ ̃ ̃ Motivation Simulation Diagnosis Workfmow Example Testing procedure for many parameters ▶ Same procedure as before ▶ Each of the 𝑞 (one-dimensional) elements 𝜄 can have its own 𝜄 𝑞 ∈ histogram ▶ Or pool all ranks for all 𝑞 elements 𝜄 𝑞 into a single histogram ▶ Adjust the expected binomial distribution accordingly Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example What you expect to see if everything is working Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example If your chain is badly autocorrelated Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example Regarding autocorrelation ▶ Autocorrelation prevents us from assessing if 𝑇 is working ▶ Hence, for each of the 𝑂 replications 1. Generate an MCMC chain 𝜄 1∶𝑁 of length 𝑁 ≥ 𝑀 2. If 𝐹𝑇𝑇(𝜄 1∶𝑁 ) < 𝑀 ▶ Increase the value of 𝑁 and return to step 1 3. Else ▶ Thin the chain to 𝑀 iterations, or ▶ Assign the 𝑁 rank values to 𝑀 + 1 bins (easy if 𝑁 mod (𝑀 + 1) = 0) Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example If the chain is overdispersed relative to true prior Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example If the chain is underdispersed relative to true prior Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example If the chain is biased downward relative to true prior Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example Continuous testing ▶ Primary goal ▶ Satisfy the weak claim that 𝑇 is probably not implemented incorrectly, and hence 𝜄 1∶𝑀 is (approximately) distributed according to 𝑞(𝜄|𝑧) ▶ Nice side effects from continuously testing MCMC code ▶ Understand the model’s structure and the implications of that structure for data ▶ Catch bugs as soon as they are introduced, saving time and embarassment Testing MCMC Samplers Jason M.T. Roos
𝑞(𝜄|𝑧) ̃ 𝜄 ∼ 𝜌(𝜄) 𝜄) Motivation Simulation Diagnosis Workfmow Example Workfmow 1. Write code to draw samples 2. Write code to simulate from the DGP ̃ 𝑧 ∼ 𝜌(𝑧| ̃ ▶ Congratulations: You just wrote the code for the counterfactual simulations ▶ Fun fact: ̃ 𝑧 is a sample from the model’s prior predictive distribution of 𝑧 . Does ̃ 𝑧 look reasonable compared to real data? Is it over- or under-dispersed? 3. Write the MCMC program 𝑇 to generate posterior samples from 4. Test and debug until the weak claim is satisfjed ▶ Congratulations: You just pre-empted R2 by fjguring out, e.g., that only 𝑞 − 2 fjxed effects are identifjed in your model Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example Workfmow (2) 5. You now have two sets of code for the same model, but which are very different from one another ▶ One samples from the prior and generates data ▶ The other takes data and generates posterior samples ▶ Fun fact: It is very likely that one or both of these will have bugs that you will need to fjx ▶ Funner fact: It is very unlikely that you will introduce a bug into one that exactly cancels out a bug introduced into the other ▶ Unless you copy/paste code Testing MCMC Samplers Jason M.T. Roos
𝛿 Motivation Simulation Diagnosis Workfmow Example A “simple” direct-utility model ▶ One good, standard price per unit ▶ Consumer chooses 𝑦 number of units in period 𝑢 to maximize utility subject to a budget constraint 𝑣(𝑦 𝑢 ) = 𝛽 e 𝜗 𝑢 log (𝛿𝑦 𝑢 + 1) 𝑣 0 (𝑦 𝑢 ) = 𝑁 𝑢 − 𝑞 𝑢 𝑦 𝑢 𝜗 𝑢 ∼ 𝑂(0, 𝜏 2 ) ▶ Prior and DGP implemented in R ▶ MCMC Sampler implemented in Stan ▶ Simulation and rank calculations implemented in R ( sbcrs ) Testing MCMC Samplers Jason M.T. Roos
Motivation Simulation Diagnosis Workfmow Example Sampling routines 1. Forward simulation from the DGP ▶ Generate fake (exogenous) data ▶ Sample parameters from their prior distribution ▶ Simulate choices from the DGP ▶ Loop over choice occasions 𝑢 ▶ Calculate utility for each feasible quantity 𝑦 ▶ Record the choice 𝑦 𝑢 yielding the highest utility 2. MCMC sampling ▶ Likelihood is rather messy, likely to have bugs… Testing MCMC Samplers Jason M.T. Roos
Recommend
More recommend
