Meta-Learning Contextual Bandit Exploration Amr Sharaf Hal Daum e - PowerPoint PPT Presentation

Sep 21, 2022 •243 likes •381 views

Meta-Learning Contextual Bandit Exploration Amr Sharaf Hal Daum e III University of Maryland Microsoft Research & University of Maryland amr@cs.umd.edu me@hal3.name Abstract 1 Can we learn to explore in contextual bandits? 2

Meta-Learning Contextual Bandit Exploration Amr Sharaf Hal Daum´ e III University of Maryland Microsoft Research & University of Maryland amr@cs.umd.edu me@hal3.name Abstract 1
Can we learn to explore in contextual bandits? 2
Contextual Bandits: News Display 3
Contextual Bandits: News Display NEW NEW NEW NEW 4
Contextual Bandits: News Display NEW 5
Contextual Bandits: News Display NEW Goal: Maximize Sum of Rewards 6
Training Mêlée by Imitation Access to π * at train Roll-out with π * Goal: learn π loss exploit (exploit) t-1 t … loss explore (explore) Deviation Roll-in with π Examples / Time 7
Generalization: Meta-Features - No direct dependency on the contexts x. - Features include: - Calibrated predicted probability p(a t | f t , x t ); - Entropy of the predicted probability distribution; - A one-hot encoding for the predicted action ft(x t ); - Current time step t; - Average observed rewards for each action. 8
A representative learning curve 9
Win / Loss Statistics Win statistics: each (row, column) entry shows the number of times the row algorithm won against the column, minus the number of losses. 10
Win / Loss Statistics Win statistics: each (row, column) entry shows the number of times the row algorithm won against the column, minus the number of losses. 11
Theoretical Guarantees - The no-regret property of Aggrevate can be leveraged in our meta-learning setting. - We relate the regret of the learner to the overall regret of π . - This shows that, if the underlying classifier improves su ffi ciently quickly, Mêlée will achieve sublinear regret. 12
Conclusion - Q: Can we learn to explore in contextual bandits? - A: Yes, by imitating an expert exploration policy; - Generalize across bandit problems using meta-features; - Outperform alternative strategies in most settings; - We provide theoretical guarantees. 13

Recommend

Neural Contextual Bandits with UCB-based Exploration Dongruo Zhou 1 Lihong Li 2 Quanquan Gu 1 1

Neural Contextual Bandits with UCB-based Exploration Dongruo Zhou 1 Lihong Li 2 Quanquan Gu 1 1 Department of Computer Science, UCLA 2 Google Research 1 / 49 Outline Background Contextual bandit problem Deep neural networks 2 / 49

739 views • 49 slides

Contextual Multi-armed Bandit Algorithm for Semiparametric Reward Model Gi-Soo Kim, Myunghee Cho

Contextual Multi-armed Bandit Algorithm for Semiparametric Reward Model Gi-Soo Kim, Myunghee Cho Paik Seoul National University June 13, 2019 Introduction We propose a new contextual multi-armed bandit (MAB) algorithm for the nonstationary

209 views • 20 slides

A Contextual-Bandit Approach to Personalized News Article Recommendation Lihong li, Wei Chu,

A Contextual-Bandit Approach to Personalized News Article Recommendation Lihong li, Wei Chu, John Langford, Rebort E. Schapire Presentator: Qingyun Wu News Recommendation Cycle A K-armed Bandit Formulation A gambler must decide which of

560 views • 14 slides

Meta-Reinforcement Learning of Structured Exploration Strategies Abhishek Gupta , Russell

Meta-Reinforcement Learning of Structured Exploration Strategies Abhishek Gupta , Russell Mendonca, YuXuan Liu, Pieter Abbeel, Sergey Levine Human Exploration vs Robot Exploration Human Exploration vs Robot Exploration Human Exploration vs Robot

1.2k views • 30 slides

The Epoch-Greedy Algorithm for Contextual Multi-armed Bandits Authors: John Langford, Tom Zhang

The Epoch-Greedy Algorithm for Contextual Multi-armed Bandits Authors: John Langford, Tom Zhang Presented by: Ben Flora Overview Bandit problem Contextual bandits Epoch-Greedy algorithm Overview Bandit problem Contextual

674 views • 19 slides

Meta Learning Shengchao Liu Background Meta Learning (AKA Learning to Learn) A

Meta Learning Shengchao Liu Background Meta Learning (AKA Learning to Learn) A fast-learning algorithm: quickly adapted from the source tasks to the target tasks Key terminologies Support Set & Query Set C-Way K-Shot

2k views • 41 slides

Contextual modal type theory: a foundation for meta-variables Brigitte Pientka School of

Contextual modal type theory: a foundation for meta-variables Brigitte Pientka School of Computer Science McGill University Montreal, Canada Joint work with A. Nanevski and F . Pfenning Contextual modal type theory: p.1/39 Outline

846 views • 51 slides

A Unified Contextual Bandit Framework for Long- and Short-Term Recommendations Maryam Tavakol and

A Unified Contextual Bandit Framework for Long- and Short-Term Recommendations Maryam Tavakol and Ulf Brefeld {tavakol,brefeld}@leuphana.de Skopje - Sep 21, 2017 Recommendation Tavakol & Brefeld, Leuphana University Lneburg 2/22

513 views • 29 slides

Reinforcement Learning n-armed bandit Kevin Spiteri April 21, 2015 n-armed bandit n-armed

Reinforcement Learning n-armed bandit Kevin Spiteri April 21, 2015 n-armed bandit n-armed bandit 0.9 0.5 0.1 0.9 0.5 0.1 0.0 0.0 0.0 0.0 estimate n-armed bandit n-armed bandit 0.9 0.5 0.1 0.9 0.5 0.1 0 0.0 0.0 0.0 0.0

676 views • 21 slides

A few meta learning papers Guy Gur-Ari Machine Learning Journal Club, September 2017 Meta

A few meta learning papers Guy Gur-Ari Machine Learning Journal Club, September 2017 Meta Learning Mechanisms for faster, better adaptation to new tasks Integrate prior experience with a small amount of new information

700 views • 28 slides

Multi-Arm Bandit Sutton and Barto Sutton slides and Silver 1 Multi-Arm Bandits Sutton and

Multi-Arm Bandit Sutton and Barto Sutton slides and Silver 1 Multi-Arm Bandits Sutton and Barto, Chapter 2 The simplest reinforcement learning problem The Exploration/Exploitation Dilemma Online decision-making involves a fundamental

842 views • 46 slides

Reinforcement Learning Kevin Spiteri April 21, 2015 n-armed bandit n-armed bandit 0.9 0.5

Reinforcement Learning Kevin Spiteri April 21, 2015 n-armed bandit n-armed bandit 0.9 0.5 0.1 n-armed bandit 0.9 0.5 0.1 0.0 0.0 0.0 0.0 estimate n-armed bandit 0.9 0.5 0.1 0 0.0 0.0 0.0 0.0 estimate 0 0 0 0.0 0

994 views • 84 slides

Bayesian Model-Agnostic Meta-Learning Taesup Kim* (presenter), Jaesik Yoon* Ousmane Dia,

Bayesian Model-Agnostic Meta-Learning Taesup Kim* (presenter), Jaesik Yoon* Ousmane Dia, Sungwoong Kim, Yoshua Bengio, Sungjin Ahn Model-Agnostic Meta-learning (MAML) gradient-based meta-learning framework meta-update task adaptation

886 views • 22 slides

Contextual Inquiry Take Aways Overview of Contextual Design Contextual inquiry

Contextual Inquiry Take Aways Overview of Contextual Design Contextual inquiry Interviewing techniques Contextual Design Contextual design is: An established process for analyzing tasks people do and designing technology to

456 views • 43 slides

Lecture outline - recap: policy gradient RL and how it can be used to build meta-RL algorithms -

Lecture outline - recap: policy gradient RL and how it can be used to build meta-RL algorithms - the exploration problem in meta-RL - an approach to encourage better exploration break - meta-RL as a POMDP - an approach for off-policy

640 views • 45 slides

The Meta-Learning Problem & Black-Box Meta-Learning CS 330 Logistics Homework 1 posted today,

The Meta-Learning Problem & Black-Box Meta-Learning CS 330 Logistics Homework 1 posted today, due Wednesday, September 30 Project guidelines will be posted by tomorrow. Plan for Today Transfer Learning - Problem formulation - Fine-tuning

728 views • 34 slides

Class 3: Multi-Arm Bandit Sutton and Barto, Chapter 2 Sutton slides and Silver 295, class 2 1

Class 3: Multi-Arm Bandit Sutton and Barto, Chapter 2 Sutton slides and Silver 295, class 2 1 Multi-Arm Bandits Sutton and Barto, Chapter2 The simplest reinforcement learning problem The Exploration/Exploitation Dilemma Online

457 views • 32 slides

Deep Exploration via Bootstrapped DQN Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin

Deep Exploration via Bootstrapped DQN Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van Roy Presenters : Irene Jiang, Jeremy Ma, Akshay Budhkar 1 Recap on Reinforcement Learning Environment Multi-armed bandit problem

1.75k views • 147 slides

Meta-transfer Learning for Few-shot Learning Yaoyao Liu Tianjin University and NUS School of

NUS-Tsinghua-Southampton Centre for Extreme Search Meta-transfer Learning for Few-shot Learning Yaoyao Liu Tianjin University and NUS School of Computing OUTLINE Research Background Methods Meta-transfer Learning Hard-task

455 views • 33 slides

MetaFun: Meta-Learning with Iterative Functional Updates Jin Xu, Jean-Francois Ton, Hyunjik Kim,

MetaFun: Meta-Learning with Iterative Functional Updates Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam R. Kosiorek, Yee Whye Teh 37th International Conference on Machine Learning Supervised Meta-Learning Supervised Meta-Learning Supervised

1.01k views • 66 slides

robustly combining supervised and bandit feedback Chicheng Zhang 1 ; Alekh Agarwal 1 ; Hal Daum

Warm-starting contextual bandits: robustly combining supervised and bandit feedback Chicheng Zhang 1 ; Alekh Agarwal 1 ; Hal Daum III 1,2 ; John Langford 1 ; Sahand Negahban 3 1 Microsoft Research, 2 University of Maryland, 3 Yale University

344 views • 10 slides

Me Meta Lear Learnin ing A Bri Brief Introduct ction Xiachong Feng Ou Outline

Me Meta Lear Learnin ing A Bri Brief Introduct ction Xiachong Feng Ou Outline Introduction to Meta Learning Types of Meta-Learning Models Papers: Optimization as a model for few-shot learning ICLR2017

913 views • 60 slides

On the Complexity of Best Arm Identification in Multi-Armed Bandit Models Aurlien Garivier

On the Complexity of Best Arm Identification in Multi-Armed Bandit Models Aurlien Garivier Institut de Mathmatiques de Toulouse Information Theory, Learning and Big Data Simons Institute, Berkeley, March 2015 Simple Multi-Armed Bandit

514 views • 29 slides

Meta-Learning of Structured Representation by Proximal Mapping Mao Li, Yingyi Ma, Xinhua

Meta-Learning of Structured Representation by Proximal Mapping Mao Li, Yingyi Ma, Xinhua Zhang University of Illinois at Chicago Motivation Goal of meta-learning: Extract prior structures from a set of tasks that allows efficient learning of

748 views • 15 slides