Learning to Collaborate in Markov Decision Processes Goran Radanovic - PowerPoint PPT Presentation

Mar 15, 2024 •140 likes •223 views

Learning to Collaborate in Markov Decision Processes Goran Radanovic , Rati Devidze, David C. Parkes, Adish Singla Motivation: Human-AI Collaboration Example setting Helper-AI Human Agent A1 Agent A2 Task (Best) responds Commits to to !

Learning to Collaborate in Markov Decision Processes Goran Radanovic , Rati Devidze, David C. Parkes, Adish Singla
Motivation: Human-AI Collaboration Example setting Helper-AI Human Agent A1 Agent A2 Task (Best) responds Commits to to ! " policy ! " Behavioral differences Agents have different models of the world [Dimitrakakis et al., NIPS 2017] 2
Motivation: Human-AI Collaboration Helper-AI Human Agent A1 Agent A2 Task ! # changes Humans change/adapt their behavior over Commits to over time policy ! " time. Can we utilize learning to adopt a good policy for A1 despite the changing behavior of A2, without detailing A2's learning dynamics? 3
Formal Model: Two-agent MDP • Episodic two-agent MDP with commitments • Goal: design a learning algorithm for A1 that achieves a sublinear regret – Implies near optimality for smooth MDPs Rewards and transitions are non-stationary. Agent A1 4
Experts with Double Recency Bias • Based on experts in MDPs: – Assign an experts algorithm to each state – Use ! values as experts’ losses [Even-Dar et al., NIPS 2005] • Introduce double recency bias & ',) 0 * ' = 1 ! Γ - & ',) )./ " − 1 " − % Recency windowing Recency modulation 5
Main Results (Informally) Theorem: The regret or ExpDRBias decays as !(# $%& '( )*+ , , . / ) , provided that the magnitude change of A2’s policy is !( # (1 ) . Theorem: Assume that the magnitude change of A2’s policy is Ω(1) . Then achieving a sublinear regret is at least as hard as learning parity with noise . 6
Thank you! • Visit me at the poster session! 7

Recommend

Markov Systems, Markov Decision Processes, and Dynamic Programming Andrew W. Moore Note to

Artificial Intelligence 15-381 April 5, 2007 Sequential Decision Problems & Markov Decision Processes Recap of last lecture Reasoning over time - Markov Processes - Hidden Markov Models - modeling state transitions - probability of

321 views • 16 slides

Outline Md Md Markov Markov Decision Decision Processes Processes Grid World Example

3/25/2017 Outline Md Md Markov Markov Decision Decision Processes Processes Grid World Example Markov Decision Processes MDP definition Optimal Policies Auto Racing Example CSE 415: Introduction to Artificial Intelligence

431 views • 11 slides

Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics

Lecture 14 Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Slides by Stuart Russell and Peter Norvig Markov Decision Processes Course

321 views • 29 slides

max ( | ) ( ) P s a U s preferences, must exist consistent utility function a s

The Winding Path to RL Markov Decision Processes & Decision Theory Descriptive theory of optimal Reinforcement Learning behavior Markov Decision Processes Mathematical/Algorithmic realization of Decision Theory

86 views • 8 slides

1 Markov Decision Processes Markov Decision Processes An MDP is defined by: An MDP is

Non-Deterministic Search CSE 473: Artificial Intelligence Markov Decision Processes Dieter Fox University of Washington [Slides originally created by Dan Klein & Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials are

370 views • 10 slides

Markov Decision Processes [RN2] Sec 17.1, 17.2, 17.4, 17.5 [RN3] Sec 17.1, 17.2, 17.4 CS 486/686

Markov Decision Processes [RN2] Sec 17.1, 17.2, 17.4, 17.5 [RN3] Sec 17.1, 17.2, 17.4 CS 486/686 University of Waterloo Lecture 13: February 14, 2012 Outline Markov Decision Processes Dynamic Decision Networks 2 CS486/686 Lecture

504 views • 18 slides

Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask

+ Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask Overview Intro Markov Decision Processes Reinforcement Learning Sarsa Q-learning Exploration vs Exploitation tradeoff 2

1.81k views • 146 slides

Introduction to Partially Observable Markov Decision Processes CS 886 Sequential Decision Making

Module 14 Introduction to Partially Observable Markov Decision Processes CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Markov Decision Processes MDPs: Fully Observable MDPs Decision maker

398 views • 21 slides

Markov decision processes and interval Markov chains: exploiting the connection Mingmei Teo

Markov decision processes and interval Markov chains: exploiting the connection Mingmei Teo Supervisors: Prof. Nigel Bean, Dr Joshua Ross University of Adelaide July 10, 2013 Background Intervals Markov Decision Processes Markov chains

528 views • 30 slides

Markov Decision Processes Mausam CSE 515 Operations Research Machine Graph Learning Theory

Markov Decision Processes Mausam CSE 515 Operations Research Machine Graph Learning Theory Control Markov Decision Process Economics Theory Neuroscience Robotics /Psychology Artificial Intelligence model the sequential decision

1.2k views • 45 slides

Markov Decision Processes (Slides from Mausam) Operations Research Machine Graph Learning

Markov Decision Processes (Slides from Mausam) Operations Research Machine Graph Learning Theory Control Markov Decision Process Economics Theory Neuroscience Robotics /Psychology Artificial Intelligence model the sequential

679 views • 35 slides

Model Repair for Markov Decision Model Repair for Markov Decision Model Repair for Markov

Model Repair for Markov Decision Model Repair for Markov Decision Model Repair for Markov Decision Model Repair for Markov Decision Processes Processes Processes Processes Marta Kwiatkowska Department of Computer Science, University of

524 views • 36 slides

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud Autef Motivation Robust Markov Decision Process (MDP) framework Tackle model mismatch and parameter uncertainty Previously, for state

520 views • 13 slides

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur Sep 14 and 15,

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur Sep 14 and 15, 2020 Agenda Terminology Markov Decision Process Agenda Understand definitions and notation to be used in the course. Understand definition

1.2k views • 95 slides

Q-learning 3-23-16 Markov Decision Processes (MDPs) States: S Actions A vs. A s

Q-learning 3-23-16 Markov Decision Processes (MDPs) States: S Actions A vs. A s Transition probabilities: P(s | s, a) Rewards R(s) vs. R(s,a) vs. R(s,s) Discount factor (sometimes considered part of the

446 views • 10 slides

Markov Decision Processes Lecture 3, CMU 10-403 Katerina Fragkiadaki Katerina Fragkiadaki

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Markov Decision Processes Lecture 3, CMU 10-403 Katerina Fragkiadaki Katerina Fragkiadaki Supervision for learning goal-seeking behaviors 1. Learning from

976 views • 71 slides

Module 4 Markov Processes CS 886 Sequential Decision Making and Reinforcement Learning

Module 4 Markov Processes CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Sequential Decision Making In general: exponentially large decision tree s1 a b . 9 . 1 . 2 . 8 s2 s3 s12 s13 a b a b

364 views • 12 slides

POMDPs (Ch. 17.4-17.6) Markov Decision Process Recap of Markov Decision Processes (MDPs): Know:

POMDPs (Ch. 17.4-17.6) Markov Decision Process Recap of Markov Decision Processes (MDPs): Know: - Current state (s) - Rewards for states (R(s)) Uncertain: - Result of actions (a) POMDPs Today we look at Partially Observable MDPs: Know: -

451 views • 26 slides

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur July 26, Aug

Markov Decision Processes CS60077: Reinforcement Learning Abir Das IIT Kharagpur July 26, Aug 01, 02, 08, 2019 Agenda Terminology Markov Decision Process Agenda Understand definitions and notation to be used in the course. Understand

1.13k views • 95 slides

Facilitating Testing and Debugging of Markov Decision Processes with Interactive Visualization

Facilitating Testing and Debugging of Markov Decision Processes with Interactive Visualization Sean McGregor, Hailey Buckingham, Thomas G. Dietterich, Rachel Houtman, Claire Montgomery, and Ronald Metoyer What are Markov Decision Processes

1.1k views • 65 slides

or, Learning and Planning with Markov Decision Processes 295 Seminar, Winter 2018 Rina Dechter

Reinforcement Learning or, Learning and Planning with Markov Decision Processes 295 Seminar, Winter 2018 Rina Dechter Slides will follow David Silvers, and Suttons book Goals: To learn together the basics of RL. Some lectures and classic

1.56k views • 127 slides

Processes (MDP) Prof. Kuan-Ting Lai 2020/3/20 Markov Decision Process (MDP)

Finite Markov Decision Processes (MDP) Prof. Kuan-Ting Lai 2020/3/20 Markov Decision Process (MDP) https://en.wikipedia.org/wiki/Markov_decision_process Markov Property Current state can represent all information from the past states

593 views • 33 slides

Recap: MDPs Op)mal Quan))es Markov decision processes:

Example: Grid World CS 232: Ar)ficial Intelligence Markov Decision Processes II A maze-like problem Oct 1 st , 2015 The agent lives in a

257 views • 12 slides

CS 188: Artificial Intelligence Markov Decision Processes II Instructor: Anca Dragan University

CS 188: Artificial Intelligence Markov Decision Processes II Instructor: Anca Dragan University of California, Berkeley [These slides adapted from Dan Klein and Pieter Abbeel] Recap: Defining MDPs o Markov decision processes: s o Set of states

543 views • 53 slides