Adaptive Antithetic Sampling for Variance Reduction Hongyu Ren*, - PowerPoint PPT Presentation

Adaptive Antithetic Sampling for Variance Reduction Hongyu Ren*, Shengjia Zhao*, Stefano Ermon *equal contribution

Goal Estimation of 𝜈 = 𝔽 𝑞(𝑦) [𝑔 𝑦 ] is ubiquitous in machine learning problems. Environment 𝑞 𝑦 𝑟(𝑨|𝑦) 𝑟(𝑨) 𝑞 𝑦 Reward State Action 𝐸 Real/Fake 𝑞(𝑨) 𝐻 𝑞(𝑦|𝑨) Agent 𝑞(𝑨) 𝑞 𝑦 log 𝑞(𝑦, 𝑨) 𝔽 𝑞(𝜐) ෍ 𝑠(𝑡 𝑢 , 𝑏 𝑢 ) 𝔽 𝑞 𝑦 𝔽 𝑟 𝑨|𝑦 𝔽 𝑞(𝑦) log 𝐸(𝑦) + 𝔽 𝑞 𝑨 log 1 − 𝐸 𝐻 𝑨 𝑟(𝑨|𝑦) 𝑢 Variational Autoencoder Generative Adversarial Nets Reinforcement Learning

Goal Estimation of 𝜈 = 𝔽 𝑞(𝑦) [𝑔 𝑦 ] is ubiquitous in machine learning problems. 1 i.i.d. Monte Carlo Estimation: 𝜈 ≈ 2 (𝑔 𝑦 1 + 𝑔(𝑦 2 )) 𝑦 1 , 𝑦 2 ∼ 𝑞(𝑦) 1 MC is unbiased: 𝔽 2 (𝑔 𝑦 1 + 𝑔(𝑦 2 )) = 𝜈 High variance Estimation can be far off with small sample size

Goal Estimation of 𝜈 = 𝔽 𝑞(𝑦) [𝑔 𝑦 ] is ubiquitous in machine learning problems. 1 i.i.d. Monte Carlo Estimation: 𝜈 ≈ 2 (𝑔 𝑦 1 + 𝑔(𝑦 2 )) 𝑦 1 , 𝑦 2 ∼ 𝑞(𝑦) Trivial solution: Better solution: use more samples! better sampling strategy than i.i.d.

Antithetic Sampling Don’t sample i.i.d. 𝑦 1 , 𝑦 2 ∼ 𝑞 𝑦 1 𝑞(𝑦 2 ) Sample correlated distribution 𝑦 1 , 𝑦 2 ∼ 𝑟(𝑦 1 , 𝑦 2 ) Unbiased if Goal: minimize 𝑟 𝑦 1 = 𝑞 𝑦 1 𝑔 𝑦 1 + 𝑔(𝑦 2 ) 𝑟 𝑦 2 = 𝑞(𝑦 2 ) Var 𝑟(𝑦 1 ,𝑦 2 ) 2

Example: Negative Sampling 𝑟 𝑦 1 , 𝑦 2 defined by 𝑦 2 𝑦 2 𝑞 𝑦 2 1.Sample 𝑦 1 ∼ 𝑞(𝑦) . 𝑦 1 Marginal 2.Pick 𝑦 2 = −𝑦 1 . on 𝑦 2 Marginal on 𝑦 1 𝑦 1 𝑞 𝑦 1

Example: Negative Sampling Best Case Example 𝑟 𝑦 1 , 𝑦 2 defined by 1.Sample 𝑦 1 ∼ 𝑞(𝑦) . 2.Pick 𝑦 2 = −𝑦 1 . 𝑔 𝑦 1 + 𝑔(𝑦 2 ) = 0 2 matches 𝑔 = 𝑦 3 𝐹 𝑞(𝑦) [𝑔 𝑦 ] = 0 𝑔 𝑦 1 +𝑔(𝑦 2 ) Var 𝑟(𝑦 1 ,𝑦 2 ) = 0 2 no error for a sample size of 2!

Example: Negative Sampling Worst Case Example 𝑟 𝑦 1 , 𝑦 2 defined by 1.Sample 𝑦 1 ∼ 𝑞(𝑦) . 2.Pick 𝑦 2 = −𝑦 1 . 𝑔 = 𝑦 2 𝑔 𝑦 1 = 𝑔(𝑦 2 ) , 𝑦 2 redundant 𝑔 𝑦 1 +𝑔(𝑦 2 ) Var 𝑟(𝑦 1 ,𝑦 2 ) doubles! 2

General Result Question: is there an antithetic distribution that always works better than i.i.d.? Yes: sampling without replacement is always a tiny bit better. No Free Lunch (Theorem 1): no antithetic distribution work better than sampling without replacement for every function 𝑔 .

Valid Distribution Set 𝑦 2 𝑦 2 𝑦 1 𝑦 1 𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒 : Set of distributions 𝑟(𝑦 1 , 𝑦 2 ) that satisfy 𝑟 𝑦 1 = 𝑞 𝑦 1 , 𝑟 𝑦 2 = 𝑞 𝑦 2

Variance of example functions Pick this distribution 𝑦 2 Low Variance 𝑦 1 High Variance 𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒 : Set of distributions 𝑟(𝑦 1 , 𝑦 2 ) that satisfy 𝑟 𝑦 1 = 𝑞 𝑦 1 , 𝑟 𝑦 2 = 𝑞 𝑦 2 1 = 𝑦 3 𝑔

Variance of example functions 𝑦 2 Pick this distribution 𝑦 1 Low Variance High Variance High Variance 𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒 : Set of distributions 𝑟(𝑦 1 , 𝑦 2 ) that satisfy 𝑟 𝑦 1 = 𝑞 𝑦 1 , 𝑟 𝑦 2 = 𝑞 𝑦 2 2 = 𝑓 𝑦 + 2𝑦𝑡𝑗𝑜(𝑦) 𝑔

Pick Good Distribution for a Class of Functions 𝑦 2  = 𝑔 1 , 𝑔 2 , … 𝑦 1 Low Variance High Variance on average for  on average for  𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒 : Set of distributions 𝑟(𝑦 1 , 𝑦 2 ) that satisfy 𝑟 𝑦 1 = 𝑞 𝑦 1 , 𝑟 𝑦 2 = 𝑞 𝑦 2

Pick Good Distribution for a class of functions 𝑦 2 𝑦 1 High Variance Low Variance on average on average 𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒 : Set of distributions 𝑟(𝑦 1 , 𝑦 2 ) that satisfy 𝑟 𝑦 1 = 𝑞 𝑦 1 , 𝑟 𝑦 2 = 𝑞 𝑦 2 Generalization Training Low variance for similar functions Pick a good 𝑟 for several functions

Training Objective 𝑔 𝑦 1 + 𝑔 𝑦 2 min 𝑟 𝔽 𝑔∼  Var 𝑟 𝑦 1 ,𝑦 2 2 𝑡. 𝑢. 𝑟 𝑦 1 , 𝑦 2 ∈ 𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒

Practical Training Algorithm We design 1. Parameterization for 𝒭 𝑣𝑜𝑐𝑗𝑏𝑡𝑓𝑒 via copulas. 2. A surrogate objective to optimize the variance.

Wasserstein GAN w/ gradient penalty Variance of Gradient Inception Score Inception Score Inception Score Inception Score Variance Wall Clock Time Batch Size Batch Size per Iteration Gulrajani, Ishaan, et al. "Improved training of wasserstein gans." Advances in Neural Information Processing Systems . 2017.

Importance Weighted Autoencoder Our method VS negative sampling Our method VS i.i.d. sampling Probability of Improvement Log Likelihood Improvement (higher is better) Burda, Yuri, Roger Grosse, and Ruslan Salakhutdinov. "Importance weighted autoencoders." arXiv preprint arXiv:1509.00519 (2015).

Conclusion • Define a general family of (parameterized) unbiased antithetic distribution. • Propose an optimization framework to learn the antithetic distribution based on the task at hand. • Sampling from the resulting joint distribution reduces variance at negligible computation cost. Welcome to our poster session for further discussions! Thursday 6:30-9pm @ Pacific Ballroom #205