deep reinforcement learning applications hacking
play

Deep Reinforcement Learning Applications + Hacking Arjun Chandra - PowerPoint PPT Presentation

Feb 04, 2024 •18 likes •258 views

Deep Reinforcement Learning Applications + Hacking Arjun Chandra Research Scientist Telenor Research / Telenor-NTNU AI Lab arjun.chandra@telenor.com @boelger 21 November 2017 https://join.slack.com/t/deep-rl-tutorial/signup The Plan Few

  1. Deep Reinforcement Learning Applications + Hacking Arjun Chandra Research Scientist Telenor Research / Telenor-NTNU AI Lab arjun.chandra@telenor.com @boelger 21 November 2017 https://join.slack.com/t/deep-rl-tutorial/signup

  2. The Plan Few words on applications (not exhaustive…) Games Board Games, Card Games, Video Games, VR, AR, TV Shows (IBM Watson) … growing list Robotics Thermal Soaring, Robots, Self-driving * , Autonomous Braking, etc. Embedded Systems Memory Control, HVAC, etc. Internet/Marketing Hack! Personalised Web Services, Customer Lifetime Energy Solar Panel Control, Data Centres Cloud/Telecommunications Scaling, Resource Provisioning, Channel Allocation, Self- organisation in Virtual Networks Health Treatment Planning (Diabetes, Epilepsy, Parkinson’s, etc.) Maritime Decision Support

  3. Backgammon Image credit: http://incompleteideas.net/sutton/book/the-book-2nd.html

  4. 1: piece can be hit by opponent >=2: opponent cannot land =3: single spare/free to move >3: multiple spare pieces! >=2 =3 1 >3 { { { 4 per place x 24 places 4 per place x 24 places turn #bar #o ff to move

  5. a move v( ) own simulated moves v( ) v( ) v( )

  6. a move v( ) own simulated moves v( ) v( ) v( ) TD error: v() - v()

  7. play to the end…

  8. TD-Gammon 0.0 • No Backgammon knowledge • NN, Backprop to represent and learn • Self-play TD to estimate returns • Good player beating programs with expert training and hand crafted features

  9. TD-Gammon >1.0++ simulation decision v() of simulated next moves time inform v() of move to play Simulation : -> own move given dice roll -> opponent dice roll -> opponent move • Specialised Backgammon features • NN, Backprop to represent and learn Assume opponent choses • Self-play TD and decision time search, best value move. to estimate returns • World class — impacted human play Best move given opponent’s best move is selected.

  10. 1992, 1994, 1995, 2002… NB. impacted human play, raised human caliber Combination of learnt value function and decision time search powerful!

  11. Deep RL in AlphaGo Zero Improve planning (search) and intuition (evaluation) with feedback from self-play [ zero human game data] observations win/lose/draw Zero Zero act act Game Mastering the game of Go without human knowledge , Silver et.al., Nature, Vol. 550, October 19, 2017

  12. Self-play NN training Image credit: http://incompleteideas.net/sutton/book/the-book-2nd.html

  13. Deep Net f θ p probability of taking one of 362 actions residual block v likelihood of of conv layers win/loss [39 to 79 layers] + p and v heads [2 layers, 3 layers] [Xt, Yt, Xt-1, Yt-1, …, Xt-7, Yt-7, C] historical map of stones X: 1/0 player stones Y: 1/0 opponent stones C: player, all 1 black, all 0 white

  14. Self-play to end of game NN training: learn to evaluate Self-play step: select move by simulation + evaluation Mastering the game of Go without human knowledge , Silver et.al., Nature, Vol. 550, October 19, 2017

  15. Thermal Soaring state: (local, descritised) acceleration (a z ), torque, velocity (v z ), temperature action: bank +/-, no-op reward: after step v z + Ca z goal: climb to cloud ceiling simulation https://www.onlinecontest.org/olc-2.0/gliding/flightinfo.html?flightId=1631541895 tabular SARSA Height (km) untrained trained Learning to soar in turbulent environments, Gautam Reddy et. al., PNAS 2016

  16. Memory Control scheduler is the agent state : based on contents of transaction queue, e.g. #read requests, #write requests, etc. action : activate , precharge , read , write , no-op reward : 1 for read or write, 0 otherwise goal : (max read/write ~ throughput) constraints on valid actions/state H/W implementation of SARSA http://incompleteideas.net/sutton/book/the-book-2nd.html http://incompleteideas.net/sutton/book/the-book-2nd.html Dynamic multicore resource management: A machine learning approach Martinez and Ipek, IEEE Micro, 2009

  17. Personalised Services (content/ads/o ff ers) goal #clicks policy #visits encouraging users to engage in extended #clicks interactions #visitors http://incompleteideas.net/sutton/book/the-book-2nd.html state : (per customer) (s,a,r,s’) time since last visit, tuples from the total visits, past policies last time clicked, location, sampled tuples and train interests, random forest to demographics predict return Personalized Ad Recommendation Systems action : o ff ers/ads for Life-Time Value Optimization with (fitted Q iteration reward : 1 click, 0 otherwise Guarantees. Theocharous et. al. IJCAI, 2015 ~ DQN)

  18. Solar Panel Control Solar tracking — pointing at sun enough? Missing: • di ff used radiation • reflected — ground/snow/surroundings • power consumed to reorient • shadows — foliage, clouds etc. state : panel orientation, relative location of sun OR downsampled 16X16 image actions : set of discrete orientations OR tilt forward/back/no-op reward : energy gathered at time step goal: maximise energy gathered over time Bandit-Based Solar Panel Control David Abel et. al. IAAI 2018 Improving Solar Panel Efficiency using Reinforcement Learning. David Abel et. https://github.com/david-abel/solar_panels_rl al. RLDM 2017

  19. Hack!

  20. Code Clone this repo: https://github.com/traai/drl-tutorial Go through README to set up Python environment and read through the tasks. Build on provided code/code from scratch. Use Slack for questions: https://join.slack.com/t/deep-rl-tutorial/signup

  21. Value Based (DQN)

  22. Catch fruit in basket! state : 1 for fruit, 1s for basket actions : left, right, no-op rewards +1: fruit caught -1: fruit not caught 0: otherwise goal : catch fruit (!) Simple DQN solution: https://github.com/traai/drl-tutorial/blob/master/value/dqn.py

  23. Policy Based

  24. Balance a pole! state action reward: 1 for each step goal: maximise cumulative reward https://github.com/openai/gym/wiki/CartPole-v0 Simple PG solution: https://github.com/traai/drl-tutorial/blob/master/pg/pg.py

Recommend

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Hacking Reinforcement Learning Guillem Duran Ballester Guillemdb @Miau_DB A tale about hacking

Hacking Reinforcement Learning Guillem Duran Ballester Guillemdb @Miau_DB A tale about hacking

Hacking Reinforcement Learning Guillem Duran Ballester Guillemdb @Miau_DB A tale about hacking AI-Corp Hacking RL 1. Information gathering 2. Scanning 3. Exploitation & privilege escalation 4. Maintaining access & covering tracks

960 views • 62 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Deep Representation and Reinforcement Learning Soumalya Sarkar, PhD for Anomaly Detection and

Deep Representation and Reinforcement Learning Soumalya Sarkar, PhD for Anomaly Detection and

Deep Representation and Reinforcement Learning Soumalya Sarkar, PhD for Anomaly Detection and Control Senior Research scientist, UTRC in Multi-modal Aerospace Applications May 9 @ GTC 2017 This document contains no technical data subject to the

1.09k views • 43 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides
Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Q-Learning Deep Q-Learning on Atari Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents Reinforcement Learning

938 views • 63 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell

10703 Deep Reinforcement Learning Reinforcement Learning in Humans and Animals Tom Mitchell October 29, 2018 Reading: Barto & Sutton Chapter 15 Tom Mitchell, October 2018 Outline RL in primates RL in humans Error signals and

448 views • 20 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides
Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model el U-Tr Trees Guiliang Liu, Oliver Schulte, Wang Zhu, Qingcan Li Machine Learning Lab, ECML-PKDD 2018 Presentation PROBLEM DEFINITION

372 views • 15 slides
SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Symbolic Deep Reinforcement Learning SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background Leveraging Symbolic Planning Method Experiment Conclusion Bo Liu and Future Work Auburn

1k views • 20 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 4: Q-Value based RL Animesh Garg Deep Reinforcement Learning with Double Q-learning Hado van Hasselt, Arthur Guez, David Silver Dueling Network Architectures for Deep

535 views • 24 slides
Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines? Intelligent machines must be able to adapt Deep learning helps us handle unstructured environments Reinforcement learning provides a formalism for

1.89k views • 172 slides

More recommend