learning action representations for reinforcement learning
play

Learning Action Representations for Reinforcement Learning Georgios - PowerPoint PPT Presentation

Dec 11, 2023 •287 likes •611 views

Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip Theocharous Jordan Chandak Kostas Thomas Reinforcement Learning Problem Statement Thousands of possible actions! Problem Statement Thousands of

  1. Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip Theocharous Jordan Chandak Kostas Thomas

  2. Reinforcement Learning

  3. Problem Statement Thousands of possible actions!

  4. Problem Statement Thousands of possible actions! ● Personalized tutoring systems

  5. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing

  6. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription

  7. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management

  8. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management ● Video/Songs recommendation

  9. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management ● Video/Songs recommendation ● … ● … ● Option selection

  10. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management ● Video/Songs recommendation ● … ● … ● Option selection

  11. Key Insights - Actions are not independent discrete quantities.

  12. Key Insights - Actions are not independent discrete quantities. - There is a low dimensional structure underlying their behavior pattern.

  13. Key Insights - Actions are not independent discrete quantities. - There is a low dimensional structure underlying their behavior pattern. - This structure can be learned independent of the reward .

  14. Key Insights - Actions are not independent discrete quantities. - There is a low dimensional structure underlying their behavior pattern. - This structure can be learned independent of the reward . - Instead of raw actions, agent can act in this space of behavior and feedback can be generalized to similar actions.

  15. Proposed Method

  16. Algorithm (a) Supervised learning of action representations.

  17. Algorithm (a) Supervised learning of action representations. (b) Learning internal policy with policy gradients.

  19. Results

  20. Results

  21. Results

  22. Results

  23. Real-world Applications at Adobe Photoshop HelpX Actions = 1498 tutorials Actions = 1843 tools

  24. Poster #112 Today

  26. Results (Action representations) Actual behavior of 2 12 Maze Learned representations of 2 12 actions domain actions

  27. Policy decomposition

  28. Case 1: Action representations are known - The internal policy acts in the space of action representations - Any existing policy gradient algorithm can be used to improve its local performance, independent of the mapping function.

  29. Case 2: Learning action representations - P(a|e) required to map representation to action can be learned by satisfying the earlier assumption: - We parameterize P(a|e) and P(e|s,s’) with learnable functions f and g , respectively. - Observed transition tuples are from the required distribution. - Parameters can be learned by minimizing the stochastic KL divergence. - Procedure is independent of reward .

  30. Experiments Toy Maze: - Agent in continuous state with n actuators. 2 n actions. Exponentially large action space. - - Long horizon and single goal reward. Adobe Datasets : - N-gram based multi-time step user behavior model from passive data. - Rewards defined using a surrogate objective. - Photoshop tool recommendation ( 1843 tools) - HelpX tutorial recommendation ( 1498 tutorials)

  31. Advantages - Exploits structure in space of actions. - Quick generalization of feedback to similar actions. - Less parameters updated using high variance policy gradients. - Drop-in extension for existing policy gradient algorithms.

Recommend

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

212 views • 20 slides
Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile

Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile

Reinforcement learning with restrictions on the action set Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile Joint work with Mathieu Faure (AMSE-GREQAM) Reinforcement learning with restrictions on the

976 views • 68 slides
Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

1 Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5 Environment Dynamics Gillian Hayes Components of a Problem 18th January 2007 Values and Action Values, V and Q Optimal Policies

109 views • 7 slides
Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

1 Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an action Lecture 2 We get a reward Gillian Hayes Our aim is to learn a policy what action to choose in what state to get maximum reward 11th

641 views • 5 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Global plan Reinforcement learning I: prediction classical conditioning

Global plan Reinforcement learning I: prediction classical conditioning

Global plan Reinforcement learning I: prediction classical conditioning dopamine Reinforcement learning II: Reinforcement learning II: dynamic programming; action selection Pavlovian misbehaviour vigor

584 views • 46 slides
Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

MACHINE LEARNING 2012 MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING 2012 Reinforcement Learning (RL) Supervised Learning Unsupervised

777 views • 51 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks to Joelle Pineau for presenting our paper - July, 2012 1 Authors Alborz Geramifard Stefanie Tellex David Wingate Nicholas Roy Jonathan How 2

491 views • 25 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing Process for bringing together a group of people with varied levels of skills and experience (but with influence, passion, etc. to impact an

191 views • 5 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David Silver, Dan Klein, Mausam, Dan Weld Reinforcement Learning At each time step t : Agent executes an action A t Environment emits a reward Rt

1.1k views • 71 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views • 29 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *, Yonathan Efroni* and Shie Mannor *equal contribution Poster #272 Action Robust Reinforcement Learning and Applications in Continuous Control Robust MDPs

502 views • 8 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control

Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control

Guidelines for Action Space Definition in Reinforcement Learning-based Traffic Signal Control Systems Maxime Trca, Julian Garbiso, Dominique Barth October 15, 2020 Outline I - Reinforcement Learning Applied to Traffic Signal Control II -

189 views • 16 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides

More recommend