cs885 reinforcement learning lecture 12 june 8 2018
play

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep - PowerPoint PPT Presentation

Mar 16, 2023 •6 likes •116 views

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Recurrent neural networks Long short term memory (LSTM) networks Deep

  1. CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1

  2. Outline • Recurrent neural networks – Long short term memory (LSTM) networks • Deep recurrent Q-networks University of Waterloo CS885 Spring 2018 Pascal Poupart 2

  3. ParEal Observability • Hidden Markov model – Initial state distribution: Pr($ % ) – Transition probabilities: Pr($ '() |$ ' ) – Observation probabilities: Pr(+ ' |$ ' ) • Belief monitoring ∝ Pr + ' $ ' ∑ / 012 Pr $ ' $ '3) Pr($ '3) |+ )..'3) ) Pr $ ' + )..' s 0 s 1 s 2 s 4 s 3 o 1 o 2 o 3 o 4 o 5 University of Waterloo CS885 Spring 2018 Pascal Poupart 3

  4. Recurrent Neural Network (RNN) • In RNNs, outputs can be fed back to the network as inputs, crea:ng a recurrent structure • HMMs can be simulated and generalized by RNNs • RNNs can be used for belief monitoring ! " : vector of observa:ons # " : belief state University of Waterloo CS885 Spring 2018 Pascal Poupart 4

  5. Training • Recurrent neural networks are trained by backpropaga6on on the unrolled network – E.g. backpropaga6on through 6me • Weight sharing: – Combine gradients of shared weights into a single gradient • Challenges: – Gradient vanishing (and explosion) – Long range memory – Predic6on driF University of Waterloo CS885 Spring 2018 Pascal Poupart 5

  6. Long Short Term Memory (LSTM) • Special gated structure to control memorization and forgetting in RNNs • Mitigate gradient vanishing • Facilitate long term memory University of Waterloo CS885 Spring 2018 Pascal Poupart 6

  7. Unrolled long short term memory &'( $ &'( # &'( % output output output X X X gate gate gate ℎ " ℎ $ ℎ % ℎ # X X X X X X forget forget forget gate gate gate input input input )* # )* $ )* % gate gate gate University of Waterloo CS885 Spring 2018 Pascal Poupart 7

  8. Deep Recurrent Q-Network • Hausknecht and Stone (2016) – Atari games • TransiBon model – LSTM network • ObservaBon model – ConvoluBonal network image image University of Waterloo CS885 Spring 2018 Pascal Poupart 8

  9. Deep Recurrent Q-Network Ini@alize weights ! and " ! at random in [−1,1] Observe current state ( Loop Execute policy for en@re episode Add episode ( ) * , + * , ) , , + , , ) - , + - , … , ) / , + / ) to experience buffer Sample episode from buffer Ini@alize ℎ 1 For 2 = 1 @ll the end of the episode do 4566 4! = 7 9 ! :;; ! (= ) *..? ), = + ? − ̂ B − 4N ! OPP ! (= Q J..H ), = G H C max G HIJ Q " L :;; " ! (= ) *..?M* ), = + ?M* 8 4! = Update weights: ! ← ! − S 4566 4! Every T steps, update target: " ! ← ! University of Waterloo CS885 Spring 2018 Pascal Poupart 9

  10. Deep Recurrent Q-Network Initialize weights ! and " ! at random in [−1,1] Observe current state ( Loop Execute policy for entire episode Add episode ( ) * , + * , ) , , + , , ) - , + - , … , ) / , + / ) to experience buffer Sample episode from buffer Initialize ℎ 1 For 2 = 1 till the end of the episode do 4566 4! = 7 9 ! :;; ! (ℎ =>* ? ) = ), ? + = − ̂ B − 4L ! MNN ! (O PQR ? S P ), ? G C max G H Q " J :;; " ! ℎ =>* ? ) = ? ) =K* , ? + =K* 8 4! ? ℎ = ← :;; " ! (ℎ =>* , ? ) = ) 4566 Update weights: ! ← ! − U 4! Every V steps, update target: " ! ← ! University of Waterloo CS885 Spring 2018 Pascal Poupart 10

  11. Results Flickering games (missing observaBons) University of Waterloo CS885 Spring 2018 Pascal Poupart 11

Recommend

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and Wright, Chapter 4] University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Op#miza#on in ML It is common to formulate ML problems as optimization

210 views • 7 slides
CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put]

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put]

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put] Sec. 11.1-11.3 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Hierarchical RL Hierarchy of goals Reach and actions in Destination

201 views • 8 slides
CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Minimax search Evaluation functions Alpha-beta pruning University

662 views • 26 slides
CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Environment dynamics Stochastic processes Markovian assumption

156 views • 14 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1,

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1,

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1, [Sze] Chapter 1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Introduction to Reinforcement Learning Course website and

424 views • 14 slides
CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7,

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7,

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7, [Sze] Sec. 4.2.1-4.2.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Exploration/exploitation tradeoff Regret

228 views • 20 slides
CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7,

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7,

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7, [Sze] Sec. 4.3.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Value Function Approximation Linear approximation

178 views • 16 slides
CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Quick recap Markov Decision Processes: value iteration ( " + * ,- Pr "

624 views • 18 slides
CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar]

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar]

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar] Sec. 2.9 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Bayesian bandits Thompson sampling Contextual bandits

464 views • 22 slides
CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar] Chap. 3, [Sze] Chap. 2, [RusNor] Sec. 17.1-17.2, 17.4, [Put] Chap. 2, 4, 5 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Markov

509 views • 17 slides
Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides
CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., & Ermon, S. (2016). Generative adversarial

464 views • 14 slides
Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885 Reinforcement Learning Paper by: P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou (Sensors 2019) Presented by: Neel Bhatt Context and

286 views • 14 slides
FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides
Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Machine Learning Summer School in Algiers Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is reinforcement learning? a way of programming agents by reward and punishment without needing to specify

1.15k views • 93 slides
Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist

Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist

Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist @ ProNav Technologies (www.pronavigator.ai) University of Waterloo CS885 Spring 2018 Pascal

703 views • 31 slides
Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides
Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

580 views • 55 slides

More recommend