CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov - PowerPoint PPT Presentation

Nov 20, 2022 •120 likes •201 views

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put] Sec. 11.1-11.3 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Hierarchical RL Hierarchy of goals Reach and actions in Destination

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put] Sec. 11.1-11.3 University of Waterloo CS885 Spring 2018 Pascal Poupart 1
Hierarchical RL • Hierarchy of goals Reach and actions in Destination autonomous driving Reach A Reach B Reach C Turn Overtake Stop Park Break Gas Steering • Theory: Semi-Markov Decision Processes University of Waterloo CS885 Spring 2018 Pascal Poupart 2
Semi-Markov Process • Definition – Set of States: ! – Transition dynamics: Pr $ % , ' $ = Pr $ % $ Pr ' $ where ' indicates the time to transition • Semi-Markovian: – Next state depends only on current state – Time spent in each state varies '′′ ' '′ $ $′ $′′ $′′′ University of Waterloo CS885 Spring 2018 Pascal Poupart 3
Semi-Markov Decision Process • Definition – Set of states: ! – Set of actions: " – Transition model: Pr(& ' ,)|&,+) – Reward model: - &,+ = /[1|&,+] – Discount factor: 0 ≤ 5 ≤ 1 • discounted: 5 < 1 undiscounted: 5 = 1 – Horizon (i.e., # of time steps): ℎ • Finite horizon: ℎ ∈ ℕ infinite horizon: ℎ = ∞ • Goal: find optimal policy University of Waterloo CS885 Spring 2018 Pascal Poupart 4
Example from Queuing Theory • Consider a retail store with two queues: – Customer service queue – Cashier queue • Semi-Markov decision process – State: ! = ($ % , $ ' ) where $ ) = # of customers in queue + – Action: , ∈ {1,2} (i.e., serve customer in queue 1 or 2) – Transition model: distribution over arrival and service times for customers in each queue. – Reward model: expected revenue of each serviced customer – expected cost associated with waiting times – Discount factor: 0 ≤ 4 < 1 – Horizon (i.e., # of time steps): ℎ = ∞ University of Waterloo CS885 Spring 2018 Pascal Poupart 5
Value Function and Policy • Objective: ! " # = ∑ & ' ( ) * + # ( ) ,-(# ( ) ) – Where 0 & = 1 2 + 1 4 + ⋯+ 1 & – Optimal policy: - ∗ such that ! " ∗ # ≥ ! " # ∀#,- • Bellman’s equation: ! ∗ # = max Pr # J ,1 #,C ' G ! ∗ (# J ) B + #,C + D E F ,G • Q-learning update: K #,C ← K #,C + M N + ' G max B F K # J ,C J − K(#,C) University of Waterloo CS885 Spring 2018 Pascal Poupart 6
Option Framework • Semi-Markov decision process where actions are options (temporally extended sub-policies) • Let ! be an option with sub-policy " and terminal states # $%& ∀( )*+ ∈ # $%& : Pr ( )*+ , 0 ( ) , ! = +CB Pr ( )*@ ( )*@CB , " ( )*@CB ∑ 3 456:45896 ∉ ; <=> ∏ @AB + F ∑ 3 456 Pr ( )*B ( ) , " ( ) D ( ) , !, ( )*+ , 0 = D ( ) , " ( ) +⋯F∑ 3 458 Pr ( )*+ ( )*+CB ,"(( )*+CB ) D ( )*B ," ( )*B D ( )*+ ," ( )*+ … University of Waterloo CS885 Spring 2018 Pascal Poupart 7
Option Framework • Bellman’s equation: ! ∗ # = max Pr # 0 ,1 #,2 [4 #,2,# 0 ,1 + 6 - ! ∗ (# 0 )] ( ) * + ,- • Q-learning update: : #,2 ← : #,2 + < = - + 6 - max ( + : # 0 ,2 0 − :(#,2) - 6 @ C where = - = ∑ @AB @ University of Waterloo CS885 Spring 2018 Pascal Poupart 8

Recommend

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and Wright, Chapter 4] University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Op#miza#on in ML It is common to formulate ML problems as optimization

210 views • 7 slides

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Minimax search Evaluation functions Alpha-beta pruning University

662 views • 26 slides

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Recurrent neural networks Long short term memory (LSTM) networks Deep

116 views • 11 slides

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Environment dynamics Stochastic processes Markovian assumption

156 views • 14 slides

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1,

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1, [Sze] Chapter 1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Introduction to Reinforcement Learning Course website and

424 views • 14 slides

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7,

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7, [Sze] Sec. 4.2.1-4.2.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Exploration/exploitation tradeoff Regret

228 views • 20 slides

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7,

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7, [Sze] Sec. 4.3.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Value Function Approximation Linear approximation

178 views • 16 slides

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Quick recap Markov Decision Processes: value iteration ( " + * ,- Pr "

624 views • 18 slides

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar]

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar] Sec. 2.9 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Bayesian bandits Thompson sampling Contextual bandits

464 views • 22 slides

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar] Chap. 3, [Sze] Chap. 2, [RusNor] Sec. 17.1-17.2, 17.4, [Put] Chap. 2, 4, 5 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Markov

509 views • 17 slides

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., & Ermon, S. (2016). Generative adversarial

464 views • 14 slides

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885 Reinforcement Learning Paper by: P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou (Sensors 2019) Presented by: Neel Bhatt Context and

286 views • 14 slides

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides

Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Machine Learning Summer School in Algiers Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is reinforcement learning? a way of programming agents by reward and punishment without needing to specify

1.15k views • 93 slides

Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist

Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist @ ProNav Technologies (www.pronavigator.ai) University of Waterloo CS885 Spring 2018 Pascal

703 views • 31 slides

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides

Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

580 views • 55 slides