reinforcement learning through the optimization lens
play

reinforcement learning through the optimization lens Benjamin Recht - PowerPoint PPT Presentation

Aug 13, 2023 •280 likes •1.03k views

reinforcement learning through the optimization lens Benjamin Recht University of California, Berkeley trustable, scalable, predictable Control Theory ! Reinforcement Learning is the study of how to use past data to enhance the future

  1. reinforcement learning through the optimization lens Benjamin Recht University of California, Berkeley

  2. trustable, scalable, predictable

  3. Control Theory ! Reinforcement Learning is the study of how to use past data to enhance the future manipulation of a dynamical system

  4. Disciplinary Biases AE/CE/EE/ME CS Reinforcement Control Theory Learning RL Control continuous discrete model action data action IEEE Transactions Science Magazine

  5. Disciplinary Biases AE/CE/EE/ME CS Reinforcement Today’s talk will try to unify these camps and point Control Theory Learning out how to merge their perspectives. RL Control continuous discrete model action data action IEEE Transactions Science Magazine

  6. Main research challenge: What are the fundamental limits of learning systems that interact with the physical environment? How well must we understand a system in order to control it? •statistical learning theory theoretical •robust control theory foundations •core optimization

  7. Control theory is the study of dynamical systems with inputs y u x t + 1 = Ax t + Bu t G y t = Cx t + Du t x t Simplest case of such systems are linear systems x t is called the state , and the dimension of the state is called the degree, d. u t is called the input , and the dimension is p. y t is called the output , and the dimension is q. For today, will only consider C = I , D = 0 ( x t observed)

  8. Reinforcement e t e r c Learning s d i Control theory is the study of dynamical systems with inputs ^ y u p ( x t + 1 | past ) = p ( x t + 1 | x t , u t ) G p ( y t | past ) = p ( y t | x t , u t ) x t Simplest example: Partially Observed Markov Decision Process (POMDP) x t is the state , and it takes values in [d] u t is called the input , and takes values in [p]. y t is called the output , and takes values in [q]. For today, will only consider when x t observed (MDP).

Recommend

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC

Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC

Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC San Jose, March 2019 Contents Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev

596 views • 45 slides
Device Placement Optimization using Reinforcement Learning By Mirhoseini et al. Shyam Tailor

Device Placement Optimization using Reinforcement Learning By Mirhoseini et al. Shyam Tailor

Device Placement Optimization using Reinforcement Learning By Mirhoseini et al. Shyam Tailor 21/11/18 1 The Problem machine. website. Figure from TensorFlow well. e.g. Scotch [3] do not work too Previous automated approaches

624 views • 19 slides
Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement Learning Quentin Cappart , Emmanuel Goutierre, David Bergman, Louis-Martin Rousseau 1 Research question Bounding mechanisms are critical in the

449 views • 24 slides
Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device Placement A. Mirhoseini, Hieu Pham, A. Goldie et al November 2019 Problem Background Tensorflow allows user to place operators on different devices to

371 views • 15 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini et al. (Google, ICML

Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini et al. (Google, ICML

Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini et al. (Google, ICML 17) Presented by: Stella Lau 21 November 2017 Motivation Problem Neural networks are large heterogeneous environment Which operations go

421 views • 18 slides
Reinforcement Learning Emma Brunskill Stanford University Winter 2019 Midterm Review

Reinforcement Learning Emma Brunskill Stanford University Winter 2019 Midterm Review

Reinforcement Learning Emma Brunskill Stanford University Winter 2019 Midterm Review Reinforcement Learning Involves Optimization Delayed consequences Generalization Exploration Learning Objectives Define the key features

1.23k views • 54 slides
Decentralized Stochastic Approximation, Optimization, and Multi-Agent Reinforcement Learning

Decentralized Stochastic Approximation, Optimization, and Multi-Agent Reinforcement Learning

Decentralized Stochastic Approximation, Optimization, and Multi-Agent Reinforcement Learning Justin Romberg, Georgia Tech ECE CAMDA/TAMIDS Seminar, Texas A& M College Station, Texas Streaming live from Atlanta, Georgia March 16, 2020

689 views • 64 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Deep Reinforcement Learning John Schulman 1 MLSS, May 2016, Cadiz 1 Berkeley Artificial

Deep Reinforcement Learning John Schulman 1 MLSS, May 2016, Cadiz 1 Berkeley Artificial

Deep Reinforcement Learning John Schulman 1 MLSS, May 2016, Cadiz 1 Berkeley Artificial Intelligence Research Lab Agenda Introduction and Overview Markov Decision Processes Reinforcement Learning via Black-Box Optimization Policy Gradient

865 views • 64 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
Module 8 Linear Programming CS 886 Sequential Decision Making and Reinforcement Learning

Module 8 Linear Programming CS 886 Sequential Decision Making and Reinforcement Learning

Module 8 Linear Programming CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Policy Optimization Value and policy iteration Iterative algorithms that implicitly solve an optimization problem Can

423 views • 16 slides
Police Patrol Optimization With Geospatial Deep Reinforcement Learning Presenter: Daniel Wilson

Police Patrol Optimization With Geospatial Deep Reinforcement Learning Presenter: Daniel Wilson

Police Patrol Optimization With Geospatial Deep Reinforcement Learning Presenter: Daniel Wilson Other Contributors: Orhun Aydin Omar Maher Mansour Raad Before we begin I am not a criminologist! We are working with a police department to

711 views • 37 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini, Hieu Pham, Quoc V.

Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini, Hieu Pham, Quoc V.

Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, Jefg Dean 1 / 21 What is device placement Consider a

450 views • 21 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi Willem van de Meent Northeastern University [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188

145 views • 10 slides

More recommend