csc 411 lecture 19 reinforcement learning
play

CSC 411: Lecture 19: Reinforcement Learning Class based on Raquel - PowerPoint PPT Presentation

Sep 21, 2022 •189 likes •587 views

CSC 411: Lecture 19: Reinforcement Learning Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto April 3, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 1 / 39

  1. CSC 411: Lecture 19: Reinforcement Learning Class based on Raquel Urtasun & Rich Zemel’s lectures Sanja Fidler University of Toronto April 3, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 1 / 39

  2. Today Learn to play games Reinforcement Learning [pic from: Peter Abbeel] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 2 / 39

  3. Playing Games: Atari https://www.youtube.com/watch?v=V1eYniJ0Rnk Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 3 / 39

  4. Playing Games: Super Mario https://www.youtube.com/watch?v=wfL4L_l4U9A Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 4 / 39

  5. Making Pancakes! https://www.youtube.com/watch?v=W_gxLKSsSIE Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 5 / 39

  6. Reinforcement Learning Resources RL tutorial – on course website Reinforcement Learning: An Introduction , Sutton & Barto Book (1998) Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 6 / 39

  7. What is Reinforcement Learning? [pic from: Peter Abbeel] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 7 / 39

  8. Reinforcement Learning Learning algorithms differ in the information available to learner ◮ Supervised: correct outputs ◮ Unsupervised: no feedback, must construct measure of good output ◮ Reinforcement learning More realistic learning scenario: ◮ Continuous stream of input information, and actions ◮ Effects of action depend on state of the world ◮ Obtain reward that depends on world state and actions ◮ not correct response, just some feedback Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 8 / 39

  9. Reinforcement Learning [pic from: Peter Abbeel] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 9 / 39

  10. Example: Tic Tac Toe, Notation Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 10 / 39

  11. Example: Tic Tac Toe, Notation Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 11 / 39

  12. Example: Tic Tac Toe, Notation Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 12 / 39

  13. Example: Tic Tac Toe, Notation Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 13 / 39

  14. Formulating Reinforcement Learning World described by a discrete, finite set of states and actions At every time step t, we are in a state s t , and we: ◮ Take an action a t (possibly null action) ◮ Receive some reward r t +1 ◮ Move into a new state s t +1 An RL agent may include one or more of these components: ◮ Policy π : agents behaviour function ◮ Value function: how good is each state and/or action ◮ Model: agent’s representation of the environment Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 14 / 39

  15. Policy A policy is the agent’s behaviour. It’s a selection of which action to take, based on the current state Deterministic policy: a = π ( s ) Stochastic policy: π ( a | s ) = P [ a t = a | s t = s ] [Slide credit: D. Silver] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 15 / 39

  16. Value Function Value function is a prediction of future reward Used to evaluate the goodness/badness of states Our aim will be to maximize the value function (the total reward we receive over time): find the policy with the highest expected reward By following a policy π , the value function is defined as: r t + γ r t +1 + γ 2 r t +2 + · · · V π ( s t ) = γ is called a discount rate, and it is always 0 ≤ γ ≤ 1 If γ close to 1, rewards further in the future count more, and we say that the agent is “farsighted” γ is less than 1 because there is usually a time limit to the sequence of actions needed to solve a task (we prefer rewards sooner rather than later) [Slide credit: D. Silver] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 16 / 39

  17. Model The model describes the environment by a distribution over rewards and state transitions: P ( s t +1 = s ′ , r t +1 = r ′ | s t = s , a t = a ) We assume the Markov property: the future depends on the past only through the current state Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 17 / 39

  18. Maze Example Rewards: − 1 per time-step Actions: N, E, S, W States: Agent’s location [Slide credit: D. Silver] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 18 / 39

  19. Maze Example Arrows represent policy π ( s ) for each state s [Slide credit: D. Silver] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 19 / 39

  20. Maze Example Numbers represent value V π ( s ) of each state s [Slide credit: D. Silver] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 20 / 39

  21. Example: Tic-Tac-Toe Consider the game tic-tac-toe: ◮ reward: win/lose/tie the game (+1 / − 1 / 0) [only at final move in given game] ◮ state: positions of X’s and O’s on the board ◮ policy: mapping from states to actions ◮ based on rules of game: choice of one open position ◮ value function: prediction of reward in future, based on current state In tic-tac-toe, since state space is tractable, can use a table to represent value function Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 21 / 39

  22. RL & Tic-Tac-Toe Each board position (taking into account symmetry) has some probability Simple learning process: ◮ start with all values = 0.5 ◮ policy: choose move with highest probability of winning given current legal moves from current state ◮ update entries in table based on outcome of each game ◮ After many games value function will represent true probability of winning from each state Can try alternative policy: sometimes select moves randomly (exploration) Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 22 / 39

  23. Basic Problems Markov Decision Problem (MDP): tuple ( S , A , P , γ ) where P is P ( s t +1 = s ′ , r t +1 = r ′ | s t = s , a t = a ) Standard MDP problems: 1. Planning: given complete Markov decision problem as input, compute policy with optimal expected return [Pic: P. Abbeel] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 23 / 39

  24. Basic Problems Markov Decision Problem (MDP): tuple ( S , A , P , γ ) where P is P ( s t +1 = s ′ , r t +1 = r ′ | s t = s , a t = a ) Standard MDP problems: 1. Planning: given complete Markov decision problem as input, compute policy with optimal expected return 2. Learning: We don’t know which states are good or what the actions do. We must try out the actions and states to learn what to do [P. Abbeel] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 24 / 39

  25. Example of Standard MDP Problem 1. Planning: given complete Markov decision problem as input, compute policy with optimal expected return 2. Learning: Only have access to experience in the MDP, learn a near-optimal strategy Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 25 / 39

  26. Example of Standard MDP Problem 1. Planning: given complete Markov decision problem as input, compute policy with optimal expected return 2. Learning: Only have access to experience in the MDP, learn a near-optimal strategy We will focus on learning, but discuss planning along the way Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 26 / 39

  27. Exploration vs. Exploitation If we knew how the world works (embodied in P ), then the policy should be deterministic ◮ just select optimal action in each state Reinforcement learning is like trial-and-error learning The agent should discover a good policy from its experiences of the environment Without losing too much reward along the way Since we do not have complete knowledge of the world, taking what appears to be the optimal action may prevent us from finding better states/actions Interesting trade-off: ◮ immediate reward (exploitation) vs. gaining knowledge that might enable higher future reward (exploration) Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 27 / 39

  28. Examples Restaurant Selection ◮ Exploitation: Go to your favourite restaurant ◮ Exploration: Try a new restaurant Online Banner Advertisements ◮ Exploitation: Show the most successful advert ◮ Exploration: Show a different advert Oil Drilling ◮ Exploitation: Drill at the best known location ◮ Exploration: Drill at a new location Game Playing ◮ Exploitation: Play the move you believe is best ◮ Exploration: Play an experimental move [Slide credit: D. Silver] Urtasun, Zemel, Fidler (UofT) CSC 411: 19-Reinforcement Learning April 3, 2016 28 / 39

Recommend

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides
Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

581 views • 55 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: UCB selects the arm

Refresh Your Understanding: Multi-armed Bandits Select all that are true: UCB selects the arm

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

758 views • 62 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Refresh Your Knowledge Fast RL Part II The prior over arm 1 is Beta(1,2) (left) and arm 2 is a

Refresh Your Knowledge Fast RL Part II The prior over arm 1 is Beta(1,2) (left) and arm 2 is a

Lecture 13: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With a few slides derived from David Silver Lecture 13: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

762 views • 40 slides
Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

719 views • 40 slides
Introduction CSCE CSCE 496/896 496/896 Lecture 7: Lecture 7: Reinforcement Reinforcement

Introduction CSCE CSCE 496/896 496/896 Lecture 7: Lecture 7: Reinforcement Reinforcement

Introduction CSCE CSCE 496/896 496/896 Lecture 7: Lecture 7: Reinforcement Reinforcement CSCE 496/896 Lecture 7: Learning Learning Consider learning to choose actions, e.g., Stephen Scott Reinforcement Learning Stephen Scott Robot

433 views • 9 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Reinforcement learning Applied artificial intelligence (EDA132) Lecture 13 2012-04-26 Elin A.

Reinforcement learning Applied artificial intelligence (EDA132) Lecture 13 2012-04-26 Elin A.

Reinforcement learning Applied artificial intelligence (EDA132) Lecture 13 2012-04-26 Elin A. Topp Material based on course book, chapter 21 (17), and on lecture Belningsbaserad inlrning / Reinforcement learning by rjan Ekeberg,

1.39k views • 126 slides
Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13

Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13

CSE/NB 528 Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13 1 Todays Agenda: All about Learning F Unsupervised Learning Sparse Coding Predictive Coding F Supervised learning Perceptrons and

479 views • 24 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to

Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to

Lecture 1: Introduction to Reinforcement Learning Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to Reinforcement Learning Outline 1. Course Logistics 2. What is Reinforcement Learning? 3.

930 views • 67 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and

Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and

Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline What is Reinforcement Learning RL Formalism 1. Reward 2. The agent 3. The environment 4. Actions 5.

1.27k views • 64 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
CSCE 496/896 Lecture 7: Learning Stephen Scott Reinforcement Learning Introduction MDPs Q

CSCE 496/896 Lecture 7: Learning Stephen Scott Reinforcement Learning Introduction MDPs Q

CSCE 496/896 Lecture 7: Reinforcement CSCE 496/896 Lecture 7: Learning Stephen Scott Reinforcement Learning Introduction MDPs Q Learning Stephen Scott TD Learning DQN (Adapted from Paul Quint) Atari Example Go Example

637 views • 53 slides
DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

This lecture will be recorded!!! Welcome to DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall 2020 Last Lecture v What is reinforcement learning? v Difference from other AI problems v Application

1.04k views • 52 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

This lecture will be recorded!!! Welcome to DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall 2020 Last lecture v Reinforcement Learning Components Model, Value function, Policy v Model-based

944 views • 78 slides

More recommend