Multi-Arm Bandit Sutton and Barto Sutton slides and Silver 1 - PowerPoint PPT Presentation

Multi-Arm Bandit Sutton and Barto Sutton slides and Silver 1

Multi-Arm Bandits Sutton and Barto, Chapter 2 The simplest reinforcement learning problem

The Exploration/Exploitation Dilemma Online decision-making involves a fundamental choice: • Exploitation Make the best decision given current information • Exploration Gather more information The best long-term strategy may involve short-term sacrifices Gather enough information to make the best overall decisions 295, class 2 3

Examples Restaurant Selection Exploitation Go to your favourite restaurant Exploration Try a new restaurant Online Banner Advertisements Exploitation Show the most successful advert Exploration Show a different advert Oil Drilling Exploitation Drill at the best known location Exploration Drill at a new location Game Playing Exploitation Play the move you believe is best Exploration Play an experimental move 295, class 2 4

Multi‐Armed Bandit 295, class 2 6

Regret 295, class 2 10

Counting Regret 295, class 2 11

Multi-Armed Bandits Linear or Sublinear Regret Regret greedy ϵ-greedy Totalregret decaying ϵ-greedy 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1819 Time-steps If an algorithm forever explores it will have linear total regret If an algorithm never explores it will have linear total regret Is it possible to achieve sublinear total regret?

Complexity of regret 295, class 2 13

Overview • Action‐value methods – Epsilon‐greedy strategy – Incremental implementation – Stationary vs. non‐stationary environment – Optimistic initial values • UCB action selection • Gradient bandit algorithms • Associative search (contextual bandits) 295, class 2 14

Basics • Maximize total reward collected – vs learn (optimal) policy (RL) • Episode is one step • Complex function of – True value – Uncertainty – Number of time steps – Stationary vs non‐stationary? 295, class 2 15

Action‐Value Methods

Greedy Algorithms 295, class 2 17

 ‐Greedy Algorithm 295, class 2 18

A sim ple bandit algorithm

One Bandit T askfrom Figure 2.1: An example The 10‐armedTestbed bandit problem from the 10-armed testbed. The true value q(a) of each of the ten actions was selected according to a normal distribution with mean 4 zero and unit variance, and then the actual rewards were selected 3 according to a mean q(a) q unit variance normal ⇤ (3) distribution, as suggested q ⇤ (5) 2 by these gray distributions. q ⇤ (9) 1 q ⇤ (4) q ⇤ (1) Reward 0 q q ⇤ (7) ⇤ (10) distribution q q ⇤ (2) ⇤ (8) -1 q ⇤ (6) -2 Run for 1000 steps Repeat the whole -3 thing 2000 times with different bandit tasks -4 1 2 3 4 5 6 7 8 9 10 Action

 ‐Greedy Methods on the 10‐ArmedTestbed

Averaging ⟶ learning rule • T o simplify notation, let us focus on one action • We consider only its rewards, and its estimate after n+ 1 rewards: Q n = R 1 + R 2 + · · · + R n- 1 . n - 1 • How can we do this incrementally (without storing all the rewards)? • Could store a running sum and count (and divide), or equivalently:

Derivation of incremental update

Tracking a Non‐stationary Problem

Standard stochastic approximation convergence conditions

Optimistic InitialValues

Optimistic InitialValues • All methods so far depend on Q 1 ( a ) , i.e.,they are biased. Q 1 ( a ) = 0 So far we have used • Suppose we initialize the action values optimistically ( Q 1 ( a ) = 5 ), e.g., on the 10-armed testbed (with alpha = 0 . 1 ) 100% optimistic, greedy Q 1 = 5, ε = 0 80% 0 realistic, ε-greedy % 60% Q 1 = 0, ε = 0.1 Optimal 0 action 40% 20% 0% 0 200 400 600 800 1000 Plays Steps

Decaying ε‐greedy 295, class 2 29

Optimism in the face of uncertainty 295, class 2 30

Optimism in the face of uncertainty 295, class 2 31

Upper Confidence Bounds 295, class 2 32

Hoeffding’s Inequality 295, class 2 33

Calculating UCB 295, class 2 34

Upper Confidence Bound (UCB) action selection • A clever way of reducing exploration over time • Focus on actions whose estimate has large degree of uncertainty • Estimate an upper bound on the true action values • Select the action with the largest (estimated) upper bound UCB c =2 E -greedy E = 0.1 Average reward Steps

Complexity of UCB Algorithm Theorem The UCB algorithm achieves logarithmic asymptotic total regret lim L t ≤ 8 log t ∆ a t →∞ a | ∆ > 0 a

UCB vs ε‐greedy on 10‐armed bandit

UCB vs ε‐greedy on 10‐armed bandit

Gradient‐Bandit Algorithms • Let H t ( a ) be a learned preference for taking action a 100% α =0.1 80% with baseline α =0.4 % 60% Optimal α =0.1 action 40% without baseline α =0.4 20% 0% 0 250 500 750 1000 Steps

Derivation of gradient‐bandit algorithm

Summary Comparison of BanditAlgorithms

Conclusions • These are all simple methods • but they are complicated enough—we will build on them • we should understand them completely • there are still open questions • Our first algorithms that learn from evaluative feedback • and thus must balance exploration and exploitation • Our first algorithms that appear to have a goal —that learn to maximize reward by trial and error

Our first dimensions! • Problems vs Solution Methods Bandits? • Evaluative vs Instructive Problem orSolution? • Associative vs Non-associative

Problem space Single State Associative Instructive feedback Evaluative feedback

Problem space Single State Associative Instructive feedback Evaluative Bandits feedback (Function optimization)

Problem space Single State Associative Instructive Supervised feedback learning Evaluative Bandits feedback (Function optimization)

Problem space Single State Associative Instructive Supervised Averaging feedback learning Evaluative Bandits feedback (Function optimization)

Problem space Single State Associative Instructive Supervised Averaging feedback learning Associative Evaluative Bandits Search feedback (Function optimization) (Contextual bandits)