CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust - PowerPoint PPT Presentation

Nov 16, 2022 •132 likes •210 views

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and Wright, Chapter 4] University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Op#miza#on in ML It is common to formulate ML problems as optimization

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and Wright, Chapter 4] University of Waterloo CS885 Spring 2018 Pascal Poupart 1
Op#miza#on in ML • It is common to formulate ML problems as optimization problems. – Min squared error – Min cross entropy – Max log likelihood – Max discounted sum of rewards University of Waterloo CS885 Spring 2018 Pascal Poupart 2
Two important classes • Line search methods – Find a direction of improvement – Select a step length • Trust region methods – Select a trust region (analog to max step length) – Find a point of improvement in the region University of Waterloo CS885 Spring 2018 Pascal Poupart 3
Trust Region Methods • Idea: – Approximate objective ! with a simpler objective " ! $ ∗ = '()*+, - ̃ – Solve ̃ !($) $ ∗ might be in a region • Problem: The optimum 0 where " ! poorly approximates ! and therefore $ ∗ might be far from optimal 1 • Solution: restrict the search to a region where we trust " ! to approximate ! well. $ ∗ = '()*+, -∈34563789:;< ̃ – Solve ̃ !($) University of Waterloo CS885 Spring 2018 Pascal Poupart 4
Example ! " o)en chosen to be a quadra5c approxima5on of " • " # ≈ ! " # = f c + ∇" * + # − * + 1 2! # − * + 0(*)(# − *) where 3" is the gradient and 0 is the hessian • Trust region o)en chosen to be a hypersphere # − * 4 ≤ 6 University of Waterloo CS885 Spring 2018 Pascal Poupart 5
Generic Algorithm trustRegionMethod ∗ and % = 0 Initialize ! , " # Repeat % ← % + 1 ∗ = ,-./0% 1 ̃ Solve " + ∗ 3(") subject to " − " +78 9 ≤ ! If ; ∗ ∗ ) then increase ! 3 " + ≈ 3(" + else decrease ! Until convergence University of Waterloo CS885 Spring 2018 Pascal Poupart 6
Trust Region Subproblem ! " often chosen to be a quadratic approximation of " • f c + ∇" + , - − + + 1 2! - − + , 2(+)(- − +) min & subject to - − + 5 ≤ 7 • When 2 is positive semi-definite – Convex optimization – Simple and globally optimal solution • When 2 is not positive semi-definite – Non-convex optimization – Simple heuristics that guarantee improvement University of Waterloo CS885 Spring 2018 Pascal Poupart 7

Recommend

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put]

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put] Sec. 11.1-11.3 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Hierarchical RL Hierarchy of goals Reach and actions in Destination

201 views • 8 slides

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Minimax search Evaluation functions Alpha-beta pruning University

662 views • 26 slides

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Recurrent neural networks Long short term memory (LSTM) networks Deep

116 views • 11 slides

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Environment dynamics Stochastic processes Markovian assumption

156 views • 14 slides

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1,

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1, [Sze] Chapter 1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Introduction to Reinforcement Learning Course website and

424 views • 14 slides

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7,

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7, [Sze] Sec. 4.2.1-4.2.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Exploration/exploitation tradeoff Regret

228 views • 20 slides

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7,

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7, [Sze] Sec. 4.3.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Value Function Approximation Linear approximation

178 views • 16 slides

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Quick recap Markov Decision Processes: value iteration ( " + * ,- Pr "

624 views • 18 slides

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar]

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar] Sec. 2.9 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Bayesian bandits Thompson sampling Contextual bandits

464 views • 22 slides

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar] Chap. 3, [Sze] Chap. 2, [RusNor] Sec. 17.1-17.2, 17.4, [Put] Chap. 2, 4, 5 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Markov

509 views • 17 slides

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., & Ermon, S. (2016). Generative adversarial

464 views • 14 slides

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885 Reinforcement Learning Paper by: P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou (Sensors 2019) Presented by: Neel Bhatt Context and

286 views • 14 slides

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides

Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Machine Learning Summer School in Algiers Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is reinforcement learning? a way of programming agents by reward and punishment without needing to specify

1.15k views • 93 slides

Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist

Dialogue Systems & Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist @ ProNav Technologies (www.pronavigator.ai) University of Waterloo CS885 Spring 2018 Pascal

703 views • 31 slides

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides

Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

580 views • 55 slides