introduction to reinforcement learning lec 01 dynamic
play

Introduction to Reinforcement Learning LEC 01 : Dynamic Programming - PowerPoint PPT Presentation

Apr 27, 2023 •297 likes •629 views

Introduction to Reinforcement Learning LEC 01 : Dynamic Programming Professor Scott Moura University of California, Berkeley Tsinghua-Berkeley Shenzhen Institute Summer 2019 Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic

  1. Introduction to Reinforcement Learning LEC 01 : Dynamic Programming Professor Scott Moura University of California, Berkeley Tsinghua-Berkeley Shenzhen Institute Summer 2019 Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 1

  2. Motivating Example: Traveling Salesman What is the shortest path to loop through N cities? [http://www.informatik.uni-leipzig.de/ meiler] [http://www.superbasescientific.com/] Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 2

  3. Traveling Salesmen What is the shortest path to loop through N cities? 500 cities, random solution Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 3

  4. Traveling Salesmen What is the shortest path to loop through N cities? 500 cities, a better solution Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 3

  5. Traveling Salesmen What is the shortest path to loop through N cities? 500 cities, best solution Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 3

  6. When to use DP? When decisions are made in stages Sometimes, decisions cannot be made in isolation. One needs to balance immediate cost with future costs. Applications Maps. Robot navigation. Urban traffic planning. Network routing protocols. Optimal trace routing in PCBs. Optimal energy management. HR scheduling and project management. Routing of telecommunications messages. Optimal truck routing through given traffic congestion pattern. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 4

  7. Richard Bellman, Ph.D. | 1920 - 1984 University of Southern California RAND Corporation Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 5

  8. Coining “Dynamic Programming” “I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes ... The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named [Charles] Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word “programming”. I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying... Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities.” Eye of the Hurricane: An Autobiography (1984) Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 6

  9. Finite-time Formulation Discrete-time system x k + 1 = f ( x k , u k ) , k = 0 , 1 , · · · , N − 1 k : discrete time index x k : state - summarizes current configuration of system at time k u k : control - decision applied at time k N : time horizon - number of times control is applied Additive Cost N − 1 � J = c k ( x k , u k ) + c N ( x N ) k = 0 c k : instantaneous cost - instantaneous cost incurred at time k c N : final cost - incurred at time N Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 7

  10. EX 1: Inventory Control Order items to meet demand, while minimizing costs x k Items in stock, at the beginning of period k u k Items ordered & delivered immediately at the beginning of period k d k Demand of items during period k (assume deterministic) Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 8

  11. EX 1: Inventory Control Order items to meet demand, while minimizing costs x k Items in stock, at the beginning of period k u k Items ordered & delivered immediately at the beginning of period k d k Demand of items during period k (assume deterministic) Stock evolves according to x k + 1 = x k + u k − d k where negative stock corresponds to backlogged demand. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 8

  12. EX 1: Inventory Control Order items to meet demand, while minimizing costs x k Items in stock, at the beginning of period k u k Items ordered & delivered immediately at the beginning of period k d k Demand of items during period k (assume deterministic) Stock evolves according to x k + 1 = x k + u k − d k where negative stock corresponds to backlogged demand. Three types of cost: (a) r ( x k ) is penalty for positive stock (holding costs) or negative stock (shortage cost) (b) The purchasing cost c k u k , where c k is the cost per unit order at time k . (c) Terminal cost R ( x N ) for excess stock or unfulfilled orders at time N . Total cost: N − 1 � J = [ r ( x k ) + c k u k ] + R ( x N ) k = 0 Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 8

  13. EX 1: Inventory Control Order items to meet demand, while minimizing costs x k Items in stock, at the beginning of period k u k Items ordered & delivered immediately at the beginning of period k d k Demand of items during period k (assume deterministic) Stock evolves according to x k + 1 = x k + u k − d k where negative stock corresponds to backlogged demand. Three types of cost: (a) r ( x k ) is penalty for positive stock (holding costs) or negative stock (shortage cost) (b) The purchasing cost c k u k , where c k is the cost per unit order at time k . (c) Terminal cost R ( x N ) for excess stock or unfulfilled orders at time N . Total cost: N − 1 � J = [ r ( x k ) + c k u k ] + R ( x N ) k = 0 Minimize cost by proper choice of { u 0 , u 1 , · · · , u N − 1 } subject to u k ≥ 0. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 8

  14. Principle of Optimality (in words) Break the multistage decision problem into subproblems. At time step k , assume you know optimal decisions for time steps k + 1 , · · · , N − 1. Compute best solution for current time step, and pair with future decisions. Start from end. Work backwards recursively. In the words of French researcher Kaufmann: An optimal policy contains only optimal subpolicies. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 9

  15. Principle of Optimality (in math) Define V k ( x k ) as the optimal “value” from time step k to the end of the time horizon N , given the current state is x k . Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 10

  16. Principle of Optimality (in math) Define V k ( x k ) as the optimal “value” from time step k to the end of the time horizon N , given the current state is x k . Then the principle of optimality (PoO) can be written in recursive form as: V k ( x k ) = min u k { c k ( x k , u k ) + V k + 1 ( x k + 1 ) } [a.k.a. “Bellman Equation”] Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 10

  17. Principle of Optimality (in math) Define V k ( x k ) as the optimal “value” from time step k to the end of the time horizon N , given the current state is x k . Then the principle of optimality (PoO) can be written in recursive form as: V k ( x k ) = min u k { c k ( x k , u k ) + V k + 1 ( x k + 1 ) } [a.k.a. “Bellman Equation”] with the boundary condition V N ( x N ) = c N ( x N ) Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 10

  18. Principle of Optimality (in math) Define V k ( x k ) as the optimal “value” from time step k to the end of the time horizon N , given the current state is x k . Then the principle of optimality (PoO) can be written in recursive form as: V k ( x k ) = min u k { c k ( x k , u k ) + V k + 1 ( x k + 1 ) } [a.k.a. “Bellman Equation”] with the boundary condition V N ( x N ) = c N ( x N ) Admittedly awkward aspects: You solve the problem backward! You solve the problem recursively! Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 10

  19. EX 2: Shortest Path Problem 6 B F 2 5 2 1 A 5 C 4 7 3 G E 2 1 4 4 5 11 D H Let V ( i ) be the shortest path from node i to node H. Ex: V ( H ) = 0. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 11

  20. EX 2: Shortest Path Problem 6 B F 2 5 2 1 A 5 C 4 7 3 G E 2 1 4 4 5 11 D H Let V ( i ) be the shortest path from node i to node H. Ex: V ( H ) = 0. Let c ( i , j ) denote the cost of traveling from node i to node j . Ex: c ( C , E ) = 7. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 11

  21. EX 2: Shortest Path Problem 6 B F 2 5 2 1 A 5 C 4 7 3 G E 2 1 4 4 5 11 D H Let V ( i ) be the shortest path from node i to node H. Ex: V ( H ) = 0. Let c ( i , j ) denote the cost of traveling from node i to node j . Ex: c ( C , E ) = 7. c ( i , j ) + V ( j ) is cost on path from node i to j , and then from j to H along shortest path. Prof. Moura | UC Berkeley | TBSI CE 295 | LEC 01 - Dynamic Programming Slide 11

Recommend

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1

Symbolic Regression for Reinforcement Learning and Dynamic System Modeling Robert Babuka 1 Research interests Clustering for building locally linear models Reinforcement learning for continuous dynamic systems Neural

604 views • 31 slides
Dynamic Programming Talk 5 by Daniela and Christoph Content Reinforcement Learning Problem

Dynamic Programming Talk 5 by Daniela and Christoph Content Reinforcement Learning Problem

Reinforcement Learning and Dynamic Programming Talk 5 by Daniela and Christoph Content Reinforcement Learning Problem Agent-Environment Interface Markov Decision Processes Value Functions Bellman equations Dynamic Programming

1.67k views • 97 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Global plan Reinforcement learning I: prediction classical conditioning

Global plan Reinforcement learning I: prediction classical conditioning

Global plan Reinforcement learning I: prediction classical conditioning dopamine Reinforcement learning II: Reinforcement learning II: dynamic programming; action selection Pavlovian misbehaviour vigor

584 views • 46 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views • 29 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007

Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007

Introduction to Reinforcement Learning Bayesian Methods in Reinforcement Learning ICML 2007 sequential decision making under uncertainty ? How Can I ... ? Move around in the physical world (e.g. driving, navigation) Play and win a game

519 views • 21 slides
Introduction to Reinforcement Learning Lecturer: Daniel Russo Scribe: Nikhil Kotecha, Ryan

Introduction to Reinforcement Learning Lecturer: Daniel Russo Scribe: Nikhil Kotecha, Ryan

B9140 Dynamic Programming & Reinforcement Learning Lecture 7 - 10/30/17 Introduction to Reinforcement Learning Lecturer: Daniel Russo Scribe: Nikhil Kotecha, Ryan McNellis, Min-hwan Oh 0 From Previous Lecture Last time, we discussed

396 views • 7 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views • 27 slides
Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David Silver, Dan Klein, Mausam, Dan Weld Reinforcement Learning At each time step t : Agent executes an action A t Environment emits a reward Rt

1.1k views • 71 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

880 views • 48 slides
Machine Learning for NLP Reinforcement learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Reinforcement learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Reinforcement learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Introduction 2 Reinforcement learning: intuition Reinforcement learning: like learning to ride a bicycle.

1.18k views • 58 slides
Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Machine Learning Summer School in Algiers Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is reinforcement learning? a way of programming agents by reward and punishment without needing to specify

1.15k views • 93 slides
Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter

Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter

Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter 10 Christos Dimitrakakis Chalmers November 21, 2013 Christos Dimitrakakis (Chalmers) Reinforcement Learning: Approximate Dynamic Programming

587 views • 22 slides
Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to

Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to

Lecture 1: Introduction to Reinforcement Learning Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to Reinforcement Learning Outline 1. Course Logistics 2. What is Reinforcement Learning? 3.

930 views • 67 slides
Reinforcement Learning a gentle introduction & industrial application Christian Hidber

Reinforcement Learning a gentle introduction & industrial application Christian Hidber

Reinforcement Learning a gentle introduction & industrial application Christian Hidber Learning learning from children FOLIE 2 REINFORCEMENT LEARNING The game: demo FOLIE 3 REINFORCEMENT LEARNING The game: setup game engine Goal:

618 views • 59 slides
Reinforcement Learning CE417: Introduction to Artificial Intelligence Sharif University of

Reinforcement Learning CE417: Introduction to Artificial Intelligence Sharif University of

Reinforcement Learning CE417: Introduction to Artificial Intelligence Sharif University of Technology Spring 2019 Soleymani Slides have been adopted from Klein and Abdeel, CS188, UC Berkeley. Reinforcement Learning 2 Recap: MDPs } Markov

906 views • 62 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Introduction to Reinforcement Learning and Q-Learning Skyler Seto (ss3349) May 2, 2016 Skyler

Introduction to Reinforcement Learning and Q-Learning Skyler Seto (ss3349) May 2, 2016 Skyler

Reinforcement Learning and Markov Decision Process Q-Learning Q-Learning Convergence Introduction to Reinforcement Learning and Q-Learning Skyler Seto (ss3349) May 2, 2016 Skyler Seto (ss3349) Introduction to Reinforcement Learning and

565 views • 27 slides
Finite-Sample Analysis in Reinforcement Learning Mohammad Ghavamzadeh INRIA Lille Nord

Finite-Sample Analysis in Reinforcement Learning Mohammad Ghavamzadeh INRIA Lille Nord

Finite-Sample Analysis in Reinforcement Learning Mohammad Ghavamzadeh INRIA Lille Nord Europe, Team SequeL Outline Introduction to RL and DP 1 Approximate Dynamic Programming (AVI & API) 2 How does Statistical Learning Theory come to

708 views • 66 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides

More recommend