cs885 reinforcement learning lecture 8b may 25 2018
play

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and - PowerPoint PPT Presentation

Nov 24, 2023 •232 likes •464 views

CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar] Sec. 2.9 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Bayesian bandits Thompson sampling Contextual bandits

  1. CS885 Reinforcement Learning Lecture 8b: May 25, 2018 Bayesian and Contextual Bandits [SutBar] Sec. 2.9 University of Waterloo CS885 Spring 2018 Pascal Poupart 1

  2. Outline • Bayesian bandits – Thompson sampling • Contextual bandits University of Waterloo CS885 Spring 2018 Pascal Poupart 2

  3. Multi-Armed Bandits • Problem: – ! bandits with unknown average reward "($) – Which arm $ should we play at each time step? – Exploitation/exploration tradeoff • Common frequentist approaches: – & -greedy – Upper confidence bound (UCB) • Alternative Bayesian approaches – Thompson sampling – Gittins indices University of Waterloo CS885 Spring 2018 Pascal Poupart 3

  4. Bayesian Learning • Notation: – ! " : random variable for # ’s rewards – Pr ! " ; ' : unknown distribution (parameterized by ' ) – ( # = *[! " ] : unknown average reward • Idea: – Express uncertainty about ' by a prior Pr(') " , ! " , … , ! " ) based on – Compute posterior Pr('|! 0 2 4 " observed for # so far. " , ! " , … , ! samples ! 0 2 4 • Bayes theorem: " ∝ Pr ' Pr(! " , ! " , … , ! " , ! " , … , ! " |') Pr ' ! 0 2 4 0 2 4 University of Waterloo CS885 Spring 2018 Pascal Poupart 4

  5. Distributional Information • Posterior over ! allows us to estimate – Distribution over next reward " # # = - # 0! Pr " # |" Pr " # ; ! Pr ! " # , " # , … , " # , " # , … , " + + ' ) ' ) . – Distribution over 1(3) when ! includes the mean # = Pr ! " # if ! = 1(3) # , " # , … , " # , " # , … , " Pr 1(3) " ' ) + ' ) + • To guide exploration: # , " ) # , … , " + # ) ≥ 1 − = – UCB : Pr 1 3 ≤ 67890( " ' # , " ) # , … , " + # – Bayesian techniques: Pr 1 3 | " ' University of Waterloo CS885 Spring 2018 Pascal Poupart 5

  6. Coin Example • Consider two biased coins ! " and ! # $ ! " = Pr ! " = ℎ)*+ $ ! # = Pr ! # = ℎ)*+ • Problem: – Maximize # of heads in , flips – Which coin should we choose for each flip? University of Waterloo CS885 Spring 2018 Pascal Poupart 6

  7. Bernoulli Variables • ! " # , ! " $ are Bernoulli variables with domain {0,1} • Bernoulli dist. are parameterized by their mean – i.e. Pr ! " # ; - . = - . = 0 1 . Pr ! " $ ; - 2 = - 2 = 0(1 2 ) University of Waterloo CS885 Spring 2018 Pascal Poupart 7

  8. Beta distribution • Let the prior Pr($) be a Beta distribution &'() $; +, - ∝ $ /01 1 − $ 401 &'() $; 1, 1 &'() $; 2, 8 • + − 1: # of heads &'()($; 20, 80) • - − 1 : # of tails Pr($) • 6 $ = +/(+ + -) $ University of Waterloo CS885 Spring 2018 Pascal Poupart 8

  9. Belief Update • Prior: Pr # = %&'( #; *, , ∝ # ./0 1 − # 3/0 • Posterior after coin flip: Pr # ℎ&(5 ∝ Pr # Pr ℎ&(5 # ∝ # ./0 1 − # 3/0 # = # .60 /0 1 − # 3/0 ∝ %&'((#; * + 1, ,) Pr # '(:; ∝ Pr # Pr '(:; # ∝ # ./0 1 − # 3/0 (1 − #) = # ./0 1 − # (360)/0 ∝ %&'((#; *, , + 1) University of Waterloo CS885 Spring 2018 Pascal Poupart 9

  10. Thompson Sampling • Idea: – Sample several potential average rewards: . , … , - . ) for each # ! " # , … ! & # ~ Pr(!(#)|- " / – Estimate empirical average ! # = " & 0 & ∑ 34" ! 3 (#) – Execute #-56#7 . 0 ! # • Coin example . = Beta < . ; > . , ? . . , … , - – Pr !(#) - " / where > . − 1 = #ℎF#GH and ? . − 1 = #I#JKH University of Waterloo CS885 Spring 2018 Pascal Poupart 10

  11. Thompson Sampling Algorithm Bernoulli Rewards ThompsonSampling( ℎ ) " ← 0 For % = 1 to ℎ Sample * + , , … , * / (,) ~ Pr(* , ) ∀, * , ← + 6 / / ∑ 89+ * 8 (,) ∀, , ∗ ← argmax ? 6 * , Execute , ∗ and receive @ " ← " + @ Update Pr(*(, ∗ )) based on @ Return " University of Waterloo CS885 Spring 2018 Pascal Poupart 11

  12. Comparison Thompson Sampling Greedy Strategy • Action Selection • Action Selection ! ∗ = argmax ) * ! ∗ = argmax ) < + ! + ! • Empirical mean • Empirical mean + ! = , + ! = , - 9 7 * < - ∑ /0, 9 ∑ /0, 6 / + / (!) • Samples • Samples 7 … 6 7 ~ Pr(6 7 ; ;) 7 ) 6 + / ! ~ Pr(+ / (!)|6 9 , / 7 ~ Pr(6 7 ; ;) 6 / • Some exploration • No exploration University of Waterloo CS885 Spring 2018 Pascal Poupart 12

  13. Sample Size • In Thompson sampling, amount of data ! and sample size " regulate amount of exploration • As ! and " increase, # $ % becomes less stochastic, which reduces exploration . … , . ) becomes more peaked – As ! ↑ , Pr($(%)|, 0 - . … , – As " ↑ , # . ] $ % approaches 1[$(%)|, - 0 • The stochasticity of # $(%) ensures that all actions are chosen with some probability University of Waterloo CS885 Spring 2018 Pascal Poupart 13

  14. Analysis • Thompson sampling converges to best arm • Theory: – Expected cumulative regret: !(log &) – On par with UCB and ( -greedy • Practice: – Sample size ) often set to 1 University of Waterloo CS885 Spring 2018 Pascal Poupart 14

  15. Contextual Bandits • In many applications, the context provides additional information to select an action – E.g., personalized advertising, user interfaces – Context: user demographics (location, age, gender) • Actions can also be characterized by features that influence their payoff – E.g., ads, webpages – Action features: topics, keywords, etc. University of Waterloo CS885 Spring 2018 Pascal Poupart 15

  16. Contextual Bandits • Contextual bandits: multi-armed bandits with states (corresponding to contexts) and action features • Formally: – ! : set of states where each state " is defined by a vector of features # $ = (' ( $ , ' * $ , … , ' , $ ) – . : set of actions where each action a is associated with a vector of features # / = (' ( / , ' * / , … , ' 0 / ) – Space of rewards (often ℝ ) • No transition function since the states at each step are independent • Goal find policy 2: # $ → 5 that maximizes expected rewards 6 7 ", 5 = 6(7|# $ , # / ) University of Waterloo CS885 Spring 2018 Pascal Poupart 16

  17. Approximate Reward Function • Common approach: – learn approximate average reward function " #, % = ! ! "(() (where ( = (( * , ( + ) ) by regression • Linear approximation: ! " , ( = , - ( • Non-linear approximation: ! " , ( = ./01%23/4((; ,) University of Waterloo CS885 Spring 2018 Pascal Poupart 17

  18. Bayesian Linear Regression • Consider a Gaussian prior: !"# $ = & $ ', ) * + ∝ -.! − $ 0 $ 2) * • Consider also a Gaussian likelihood: !"# 2|4, $ = & 2 $ 0 4, 5 * ∝ -.! − 2 − $ 0 4 * 25 * • The posterior is also Gaussian: !"# $|2, 4 ∝ !"# $ Pr 2 4, $ ∝ -.! − $ 0 $ -.! − 2 − $ 0 4 * 2) * 25 * = &($|9, :) where 9 = 5 <* :42 and : = 5 <* 44 0 + ) <* + <> University of Waterloo CS885 Spring 2018 Pascal Poupart 18

  19. Predictive Posterior • Consider a state-action pair (" # , " % ) = " for which we would like to predict the reward ( • Predictive posterior: . / ( . 0 ", 1 2 / . 3, 4 *. )*+ (|" = ∫ = /((|1 2 " 0 3, " 0 4") • UCB: Pr ( < 1 2 " 0 3 + 9 " 0 4" > 1 − = where 9 = 1 + ln 2/= /2 ( ~ /((|1 2 " 0 3, " 0 4") • Thomson sampling: B University of Waterloo CS885 Spring 2018 Pascal Poupart 19

  20. Upper Confidence Bound (UCB) Linear Gaussian UCB( ℎ ) " ← 0 , %&' (|*, , = . ( /, 0 1 2 Repeat until 3 = ℎ Receive state 4 5 For each action 4 6 where 4 = (4 5 , 4 6 ) do 9:3';&<39<=:>3&(?) = @ 1 4 A * + 9 4 A ,4 ? ∗ ← argmax I 9:3';&<39<=:>3&(?) Execute ? ∗ and receive J " ← " + J update * and , based on 4 = (4 5 , 4 6 ∗ ) and J Return " University of Waterloo CS885 Spring 2018 Pascal Poupart 20

  21. Thompson Sampling Algorithm Linear Gaussian ThompsonSampling( ℎ ) " ← 0 ; %&' (|*, , = . ( /, 0 1 2 For 3 = 1 to ℎ Receive state 7 8 For each action 7 9 where 7 = (7 8 , 7 9 ) do Sample < = > , … , < @ (>) ~ .(B|C 1 7 D *, 7 D ,7) = E @ @ ∑ GH= < > ← < G (>) > ∗ ← argmax O E < > Execute > ∗ and receive B " ← " + B update * and , based on 7 = (7 8 , 7 9 ∗ ) and B Return " University of Waterloo CS885 Spring 2018 Pascal Poupart 21

  22. Industrial Use • Contextual bandits are now commonly used for – Personalized advertising – Personalized web content • MSN news: 26% improvement in click through rate after adoption of contextual bandits (https://www.microsoft.com/en- us/research/blog/real-world-interactive-learning- cusp-enabling-new-class-applications/) University of Waterloo CS885 Spring 2018 Pascal Poupart 22

Recommend

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Environment dynamics Stochastic processes Markovian assumption

156 views • 14 slides
CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1,

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1,

CS885 Reinforcement Learning Lecture 1a: May 2, 2018 Course Introduction [SutBar] Chapter 1, [Sze] Chapter 1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Introduction to Reinforcement Learning Course website and

424 views • 14 slides
CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7,

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7,

CS885 Reinforcement Learning Lecture 8a: May 25, 2018 Multi-armed Bandits [SutBar] Sec. 2.1-2.7, [Sze] Sec. 4.2.1-4.2.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Exploration/exploitation tradeoff Regret

228 views • 20 slides
CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7,

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7,

CS885 Reinforcement Learning Lecture 4b: May 11, 2018 Deep Q-networks [SutBar] Sec. 9.4, 9.7, [Sze] Sec. 4.3.2 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Value Function Approximation Linear approximation

178 views • 16 slides
CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and

CS885 Reinforcement Learning Lecture 14c: June 15, 2018 Trust Region Methods [Nocedal and Wright, Chapter 4] University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Op#miza#on in ML It is common to formulate ML problems as optimization

210 views • 7 slides
CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8

CS885 Reinforcement Learning Lecture 4a: May 11, 2018 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Quick recap Markov Decision Processes: value iteration ( &quot; + * ,- Pr &quot;

624 views • 18 slides
CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put]

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put]

CS885 Reinforcement Learning Lecture 15c: June 20, 2018 Semi-Markov Decision Processes [Put] Sec. 11.1-11.3 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Hierarchical RL Hierarchy of goals Reach and actions in Destination

201 views • 8 slides
CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4

CS885 Reinforcement Learning Lecture 13c: June 13, 2018 Adversarial Search [RusNor] Sec. 5.1-5.4 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Minimax search Evaluation functions Alpha-beta pruning University

662 views • 26 slides
CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Recurrent neural networks Long short term memory (LSTM) networks Deep

116 views • 11 slides
CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar]

CS885 Reinforcement Learning Lecture 2a: May 4, 2018 Intro to Markov decision processes [SutBar] Chap. 3, [Sze] Chap. 2, [RusNor] Sec. 17.1-17.2, 17.4, [Put] Chap. 2, 4, 5 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Markov

509 views • 17 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper

Neural Combinatorial Optimization With Reinforcement Learning CS885 Reinforcement Learning Paper by Bello, I., Pham, H., Le, Q. V., Norouzi, M., &amp; Bengio, S. (2016) Presented by Yan Shi Outline 1. Introduction 2. Background 3. Algorithms

966 views • 27 slides
CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., &amp; Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., &amp; Ermon, S. (2016). Generative adversarial

464 views • 14 slides
Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885

Reinforcement Learning-Based End-to-End Parking for Automatic Parking System CS885 Reinforcement Learning Paper by: P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou (Sensors 2019) Presented by: Neel Bhatt Context and

286 views • 14 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides
FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides
Dialogue Systems &amp; Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist

Dialogue Systems &amp; Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist

Dialogue Systems &amp; Reinforcement Learning Nabiha Asghar Ph.D. student @ UW Data Scientist @ ProNav Technologies (www.pronavigator.ai) University of Waterloo CS885 Spring 2018 Pascal

703 views • 31 slides
Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides
Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

580 views • 55 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: UCB selects the arm

Refresh Your Understanding: Multi-armed Bandits Select all that are true: UCB selects the arm

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

758 views • 62 slides

More recommend