Deep Reinforcement Learning [Mastering the Game of Go with Deep - PowerPoint PPT Presentation

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017

Outline • AlphaGo – Supervised Learning of Policy Networks – Reinforcement Learning of Policy Networks – Reinforcement Learning of Value Networks – Searching with Policy and Value Networks 2 CS486/686 Lecture Slides (c) 2017 P. Poupart

Game of Go • (simplified) rules: – Two players (black and white) – Players alternate to place a stone of their color on a vacant intersection. – Connected stones without any liberty (i.e., no adjacent vacant intersection) are captured and removed from the board – Winner: player that controls the largest number of intersections at the end of the game 3 CS486/686 Lecture Slides (c) 2017 P. Poupart

Computer Go Deep RL Monte Carlo Tree Search • Oct 2015: • March 2016: AlphaGo defeats Lee Sedol (9-dan) 4

Winning Strategy • Four steps: 1. Supervised Learning of Policy Networks 2. Reinforcement Learning of Policy Networks 3. Reinforcement Learning of Value Networks 4. Searching with Policy and Value Networks 5 CS486/686 Lecture Slides (c) 2017 P. Poupart

Policy Network • Train policy network to imitate Go experts based on a database of 30 million board configurations from the KGS Go Server. • Policy network: – Input: state (board configuration) – Output: distribution over actions (intersection on which the next stone will be placed) 6 CS486/686 Lecture Slides (c) 2017 P. Poupart

Supervised Learning of the Policy Network • Let be the weights of the policy network • Training: – Data: suppose is optimal in – Objective: maximize – Gradient: 𝒙 – Weight update: 7 CS486/686 Lecture Slides (c) 2017 P. Poupart

Reinforcement Learning of the Policy Network • How can we update a policy network based on reinforcements instead of the optimal action? • Let be the discounted sum of rewards in a trajectory that starts in by executing . • Gradient: 𝒙 – Intuition rescale supervised learning gradient by – Formally: see derivation in [Sutton and Barto, Reinforcement learning, Chapter 13] • Weight update: 8 CS486/686 Lecture Slides (c) 2017 P. Poupart

Reinforcement Learning of the Policy Network • In computer Go, program repeatedly plays games against its former self. • For each game • For each of turn of the game, compute – Gradient: 𝒙 – Weight update: 9 CS486/686 Lecture Slides (c) 2017 P. Poupart

Value Network � • Predict (i.e., who will win game) in each state with a value network – Input: state (board configuration) – Output: expected discounted sum of rewards 10 CS486/686 Lecture Slides (c) 2017 P. Poupart

Reinforcement Learning of Value Networks • Let be the weights of the value network • Training: – Data: where – Objective: minimize – Gradient: 𝒘 – Weight update: 11 CS486/686 Lecture Slides (c) 2017 P. Poupart

Searching with Policy and Value Networks • AlphaGo combines policy and value networks into a Monte Carlo Tree Search algorithm • Idea: construct a search tree – Node: – Edge: 12 CS486/686 Lecture Slides (c) 2017 P. Poupart

Deep Reinforcement Learning [Mastering the Game of Go with Deep - PowerPoint PPT Presentation

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

Deep Reinforcement Learning for Robotics:

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

Deep Reinforcement Learning Philipp Koehn 18 April 2019 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence: