gpu based a3c for deep reinforcement learning
play

GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING M. Babaeizadeh , , - PowerPoint PPT Presentation

Jun 17, 2023 •661 likes •1.16k views

GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING M. Babaeizadeh , , I.Frosio , S.Tyree , J. Clemons , J.Kautz University of Illinois at Urbana-Champaign, USA NVIDIA, USA An ICLR 2017 paper A github project GPU-BASED

  1. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING M. Babaeizadeh †,‡ , I.Frosio ‡ , S.Tyree ‡ , J. Clemons ‡ , J.Kautz ‡ † University of Illinois at Urbana-Champaign, USA ‡ NVIDIA, USA An ICLR 2017 paper A github project

  2. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING M. Babaeizadeh †,‡ , I.Frosio ‡ , S.Tyree ‡ , J. Clemons ‡ , J.Kautz ‡ † University of Illinois at Urbana-Champaign, USA ‡ NVIDIA, USA

  3. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING M. Babaeizadeh †,‡ , I.Frosio ‡ , S.Tyree ‡ , J. Clemons ‡ , J.Kautz ‡ † University of Illinois at Urbana-Champaign, USA ‡ NVIDIA, USA

  4. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING 4

  5. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING Learning to accomplish a task Image from www.33rdsquare.com 5

  6. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING Definitions ~R t ✓ Environment S t , R t ✓ Agent ✓ Observable status S t RL agent S t ✓ Reward R t ✓ Action a t ✓ Policy a t = p ( S t ) a t = p ( S t ) 6

  7. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING Definitions ~R t S t, R t Deep RL agent S t a t = p ( S t ) 7

  8. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING Definitions ~R t S t, R t Dp (∙) S t R 0 R 1 R 2 R 3 R 4 a t = p ( S t ) 8

  9. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING 9

  10. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING Asynchronous Advantage Actor-Critic (Mnih et al., arXiv:1602.01783v2, 2015) Dp (∙) Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master S t, R t p ’(∙) Agent 2 model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 10

  11. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING 11

  12. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING The GPU 12

  13. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING LOW OCCUPANCY (33%) LOW OCCUPANCY 13

  14. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING HIGH OCCUPANCY (100%) Batch size 14

  15. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING HIGH OCCUPANCY (100%), LOW UTILIZATION (40%) time time 15

  16. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING HIGH OCCUPANCY (100%), HIGH UTILIZATION (100%) time time 16

  17. GPU-BASED A3C FOR DEEP REINFORCEMENT LEARNING BANDWIDTH LIMITED time time 17

  18. Dp (∙) Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master S t, R t p ’ ( ∙ ) Agent 2 model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t + R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 18

  19. MAPPING DEEP PROBLEMS TO A GPU REGRESSION, CLASSIFICATION, REINFORCEMENT LEARNING … data status, reward ? 100% utilization / occupancy Pear, pear, pear, pear, … Empty, empty, … Fig, fig, fig, fig, fig, fig, labels action Strawberry, Strawberry, Strawberry, … … 19

  20. A3C Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master Agent 2 S t, R t model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 20

  21. A3C Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master Agent 2 S t, R t model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 21

  22. A3C Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master Agent 2 S t, R t model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 22

  23. A3C Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master Agent 2 S t, R t model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 23

  24. A3C Dp (∙) Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master Agent 2 S t, R t model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 24

  25. A3C Dp (∙) Agent 1 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) Master Agent 2 S t, R t p ’(∙) model R 0 R 1 R 2 R 3 R 4 a t = p (S t ) … Agent 16 S t, R t R 0 R 1 R 2 R 3 R 4 a t = p (S t ) 25

  26. GA3C GPU-based A3C 26 El Capitan big wall, Yosemite Valley

  27. GA3C (INFERENCE) a t {a t } Agent 1 prediction queue { S t } … … S t Master Agent 2 model predictors … Agent N 27

  28. GA3C (TRAINING) Agent 1 Dp (∙) Master Agent 2 model S t ,R t … … … Agent N R 4 { S t ,R t } R 0 R 1 R 2 training queue trainers 28

  29. GA3C a t {a t } Agent 1 prediction queue Dp (∙) { S t } … … S t Master Agent 2 model predictors … … … Agent N R 4 { S t ,R t } R 0 R 1 R 2 training queue trainers 29

  30. GA3C GPU-based A3C 30 El Capitan big wall, Yosemite Valley

  31. GA3C Learn how to balance GPU-based A3C 31 El Capitan big wall, Yosemite Valley

  32. GA3C: PREDICTIONS PER SECOND (PPS) a t {a t } Agent 1 prediction queue Dp (∙) { S t } … … S t Master Agent 2 model predictors … … … Agent N R 4 { S t ,R t } R 0 R 1 R 2 training queue trainers 32

  33. GA3C: TRAININGS PER SECOND (TPS) a t {a t } Agent 1 prediction queue Dp (∙) { S t } … … S t Master Agent 2 model predictors … … … Agent N R 4 { S t ,R t } R 0 R 1 R 2 training queue trainers 33

  34. AUTOMATIC SCHEDULING Balancing the system at run time ATARI Boxing ATARI Pong N P = # predictors, N T = # trainers, N A = # agents, TPS = training per seconds 34

  35. THE ADVANTAGE OF SPEED More frames = faster convergence 35

  36. LARGER DNNS For real world applications (e.g. robotics, automotive) A3C (ATARI) Others (robotics) Timothy P. Lillicrap et al., Continuous Conv 16 Conv 32 control with deep reinforcement learning, 8x8 4x4 International Conference on Learning FC 256 filters, filters, Representations, 2016. stride 4 stride 2 S. Levine et al., End-to-end training of deep ? visuomotor policies, Journal of Machine Conv 32 Learning Research, 17:1-40, 2016. Conv 32 Conv 64 8x8 4x4 4x4 filters, FC 256 filters, filters, stride 1, stride 2 stride 2 2, 3, 4 36

  37. GA3C VS. A3C*: PREDICTIONS PER SECONDS * Our Tensor Flow implementation on a CPU 10000 4x 11x 12x 20x 45x 1000 PPS A3C 100 GA3C 10 1 Small DNN Large DNN, Large DNN, Large DNN, Large DNN, stride 4 stride 3 stride 2 stride 1 37

  38. CPU & GPU UTILIZATION IN GA3C For larger DNNs 100 90 80 70 Utilization (%) 60 50 CPU % 40 GPU % 30 20 10 0 Small DNN Large DNN, Large DNN, Large DNN, Large DNN, stride 4 stride 3 stride 2 stride 1 38

  39. GA3C POLICY LAG Asynchronous playing and training {a t } Agent 1 prediction queue Dp (∙) { S t } … … S t Master Agent 2 model predictors … DNN A … … Agent N R 4 { S t ,R t } R 0 R 1 R 2 training queue trainers DNN B 39

  40. STABILITY AND CONVERGENCE SPEED Reducing policy lag through min training batch size 40

  41. GA3C (45x faster) Balancing computational resources, speed and stability. GPU-based A3C 41 El Capitan big wall, Yosemite Valley

  42. RESOURCES THEORY CODING M. Babaeizadeh, I. Frosio, S. Tyree, J. GA3C, a GPU implementation of A3C Clemons, J. Kautz, Reinforcement (open source at Learning through Asynchronous https://github.com/NVlabs/GA3C). Advantage Actor-Critic on a GPU , A general architecture to generate ICLR 2017 (available at and consume training data. https://openreview.net/forum?id=r1V GvBcxl&noteId=r1VGvBcxl). 42

  43. QUESTIONS QUESTIONS ATARI 2600 Policy lag Multiple GPUs Why TensorFlow Replay memory … Github: https://github.com/NVlabs/GA3C ICLR 2017: Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU. 43

  44. BACKUP SLIDES

  45. POLICY LAG IN GA3C Potentially large time lag between training data generation and network update 45

  46. SCORES 46

  47. Training a larger DNN 47

  48. BALANCING COMPUTATIONAL RESOURCES The actors: CPU, PCI-E & GPU Trainings per second Prediction queue size Training queue size 48

Recommend

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh Garg Deep Reinforcement Learning with Double Q-learning Hado van Hasselt, Arthur Guez, David Silver Dueling Network Architectures for Deep

535 views • 24 slides
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

212 views • 20 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides
Model Based Reinforcement Learning Katerina Fragkiadaki Model Anything the agent can use to

Model Based Reinforcement Learning Katerina Fragkiadaki Model Anything the agent can use to

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Model Based Reinforcement Learning Katerina Fragkiadaki Model Anything the agent can use to predict how the environment will respond to its actions,

826 views • 65 slides
A Robotic Auto-Focus System based on Deep Reinforcement Learning Xiaofan Yu, Runze Yu, Jingsong

A Robotic Auto-Focus System based on Deep Reinforcement Learning Xiaofan Yu, Runze Yu, Jingsong

A Robotic Auto-Focus System based on Deep Reinforcement Learning Peking University A Robotic Auto-Focus System based on Deep Reinforcement Learning Xiaofan Yu, Runze Yu, Jingsong Yang, Xiaohui Duan* School of EECS, Peking University Beijing,

340 views • 21 slides
Language Understanding for Text-based Games Using Deep Reinforcement Learning Karthik

Language Understanding for Text-based Games Using Deep Reinforcement Learning Karthik

Language Understanding for Text-based Games Using Deep Reinforcement Learning Karthik Narasimhan, Tejas Kulkarni, Regina Barzilay MIT Text-based games (State 1: The old bridge) You are standing very close to the bridges eastern

522 views • 34 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides
Calibrated Model-Based Deep Reinforcement Learning IC ML 2019 Ali Malik*, Volodymyr Kuleshov*,

Calibrated Model-Based Deep Reinforcement Learning IC ML 2019 Ali Malik*, Volodymyr Kuleshov*,

Calibrated Model-Based Deep Reinforcement Learning IC ML 2019 Ali Malik*, Volodymyr Kuleshov*, Jiaming Song, Danny Nemer, Harlan Seymour, Stefano Ermon June 13, 2019 *equal contribution Overview Importance of predictive uncertainty

388 views • 18 slides
Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Q-Learning Deep Q-Learning on Atari Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents Reinforcement Learning

938 views • 63 slides
10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell October 29, 2018 Reading: Barto & Sutton Chapter 15 Tom Mitchell, October 2018 Outline RL in primates RL in humans Error signals and

448 views • 20 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides

More recommend