Statistics and Samples in Distributional Reinforcement Learning - PowerPoint PPT Presentation

Statistics and Samples in Distributional Reinforcement Learning Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G. Bellemare, Will Dabney ICML 2019 Google Research Brain team

Distributional Reinforcement Learning Distributional RL aims to learn Distributional Bellman operator: full return distributions. Return distribution: [Bellemare et al., 2017] Distributional Bellman equation: Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Distributional Reinforcement Learning In practice , we often work with parametric approximate distributions . Non-parametric Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Distributional Reinforcement Learning In practice , we often work with parametric approximate distributions . Non-parametric Categorical [Bellemare et al., 2017] Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Distributional Reinforcement Learning In practice , we often work with parametric approximate distributions . Non-parametric Categorical [Bellemare et al., 2017] Dirac deltas [Dabney et al., 2018] Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Main Contribution: An Alternative Perspective Distributional RL algorithms learn statistical functionals of the return distribution. ● Moments, tail probabilities, expectations, etc. Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Main Contribution: An Alternative Perspective Distributional RL algorithms learn statistical functionals of the return distribution. ● Moments, tail probabilities, expectations, etc. Theory: What properties of return distributions can be learnt through dynamic programming? Algorithmic: A general framework for approximate learning of statistics. Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

A General Framework for Distributional RL Algorithms Current statistics Bellman-updated statistics Imputation strategy Imputed samples Bellman-updated distribution Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Application: Expectiles We apply this framework to learn expectiles of return distributions. New deep RL agent: Expectile Regression DQN (ER-DQN) , with improved mean performance on Atari-57 relative to QR-DQN. Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

Summary A new perspective on distributional RL Theoretical progress on what it is possible to learn A general framework for distributional RL algorithms Statistics and Samples in Distributional Reinforcement Learning — MARK ROWLAND

THANK YOU Poster #113

Statistics and Samples in Distributional Reinforcement Learning - PowerPoint PPT Presentation

Statistics and Samples in Distributional Reinforcement Learning Mark Rowland, Robert Dadashi, Saurabh Kumar, Rmi Munos, Marc G. Bellemare, Will Dabney ICML 2019 Google Research Brain team Distributional Reinforcement Learning Distributional

Statistics and Samples in Distributional Reinforcement Learning Rowland, Dadashi, Kumar, Munos,

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

A Comparative Analysis of Expected and Distributional Reinforcement Learning Clare Lyle, Pablo

Business Statistics CONTENTS Comparing two samples Comparing two unrelated samples Comparing

How to use Statas sem with small samples? New corrections for the L. R. 2 statistics and

12. Classical statistics Andrej Bogdanov Estimators X = ( X 1 , , X n ) independent samples ^

Distributional Reinforcement Learning for Efficient Exploration Hengshuai Yao Huawei Hi-Silicon

Distributional Semantics The unsupervised modeling of meaning on a large scale Tim Van de Cruys

Samples and Statistics The objective of statistical inference is to draw conclusions or make

Distributional Compositionality Intro to Distributional Semantics Raffaella Bernardi University

Optimal choice of order statistics under confidence region estimation in case of large samples cko

Single Factor Experiments Moving from comparing two samples (e.g. two formulations of mortar) to

Samples Advertising of samples and handing out samples Advertising Education and Assurance

Sim2Real Katerina Fragkiadaki So far The requirement of large number of samples for RL, only

Causal Regularization for Distributional Robustness and Replicability Peter B uhlmann Seminar

ACMS 20340 Statistics for Life Sciences Chapter 7: Samples and Observational Studies Obtaining

Statistics for Machine Learning Prof. Seungchul Lee Industrial AI Lab. Statistics and

Lecture 20 Random Samples 0/ 13 One of the most important concepts in statistics is that of a

HepSim Monte Carlo samples and their interface with detector simulations S. Chekanov (ANL),

Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth

The Role of Dimensionality Reduction in Distributional Semantics or: having fun with matrix

Automatic construction of distributional thesaurus (for multiple languages) Zheng ZHANG 1 st

10. Bayesian Statistics Andrej Bogdanov The Central Dogma of Statistics data = independent

Comparing Several Samples We are often interested in comparing measurements made under more than