reduced variance payoff estimation in adversarial bandit
play

Reduced Variance Payoff Estimation in Adversarial Bandit Problems - PowerPoint PPT Presentation

Aug 28, 2022 •99 likes •1.29k views

Improved Bandit Algorithms Reduced Variance Payoff Estimation in Adversarial Bandit Problems Levente Kocsis Csaba Szepesv ari Computer and Automation Research Institute of the Hungarian Academy of Sciences Kende u. 13-17, Budapest 1111,

  1. Improved Bandit Algorithms Universal Prediction with Expert Advice Prediction with Expert Advice Setting the goal Goal #1: Maximize total reward Very ambitious!! (too ambitious!) Goal #2: Minimize loss over the total reward of the best single expert (regret)! Goal #2.1: .. uniformly over all adversaries of interest Consequences: ⇒ There should be a single good expert for each of the adversaries of interest ⇒ We might need a large number of experts ⇒ Bounds should scale well with the # experts 1 Formally: G i , n = � n t = 1 g ( i , Y t ) , ˆ G n = � n t = 1 g ( I t , Y t ) ; we want G i , n − ˆ R n = max G n → min . i 1 Alternative is to consider tracking L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  2. Improved Bandit Algorithms Universal Prediction with Expert Advice Prediction with Expert Advice Setting the goal Goal #1: Maximize total reward Very ambitious!! (too ambitious!) Goal #2: Minimize loss over the total reward of the best single expert (regret)! Goal #2.1: .. uniformly over all adversaries of interest Consequences: ⇒ There should be a single good expert for each of the adversaries of interest ⇒ We might need a large number of experts ⇒ Bounds should scale well with the # experts 1 Formally: G i , n = � n t = 1 g ( i , Y t ) , ˆ G n = � n t = 1 g ( I t , Y t ) ; we want G i , n − ˆ R n = max G n → min . i 1 Alternative is to consider tracking L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  3. Improved Bandit Algorithms Universal Prediction with Expert Advice Prediction with Expert Advice Setting the goal Goal #1: Maximize total reward Very ambitious!! (too ambitious!) Goal #2: Minimize loss over the total reward of the best single expert (regret)! Goal #2.1: .. uniformly over all adversaries of interest Consequences: ⇒ There should be a single good expert for each of the adversaries of interest ⇒ We might need a large number of experts ⇒ Bounds should scale well with the # experts 1 Formally: G i , n = � n t = 1 g ( i , Y t ) , ˆ G n = � n t = 1 g ( I t , Y t ) ; we want G i , n − ˆ R n = max G n → min . i 1 Alternative is to consider tracking L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  4. Improved Bandit Algorithms Universal Prediction with Expert Advice Prediction with Expert Advice Setting the goal Goal #1: Maximize total reward Very ambitious!! (too ambitious!) Goal #2: Minimize loss over the total reward of the best single expert (regret)! Goal #2.1: .. uniformly over all adversaries of interest Consequences: ⇒ There should be a single good expert for each of the adversaries of interest ⇒ We might need a large number of experts ⇒ Bounds should scale well with the # experts 1 Formally: G i , n = � n t = 1 g ( i , Y t ) , ˆ G n = � n t = 1 g ( I t , Y t ) ; we want G i , n − ˆ R n = max G n → min . i 1 Alternative is to consider tracking L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  5. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  6. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Results for Adversarial Bandit Problems Theorem For any time horizon n , the expected total regret of the Exp3 algorithm is at most √ 2 2 n N ln N P . Auer et. al: “The nonstochastic multi-armed bandit problem”, SIAM Journal on Computing , 32:48–77, 2002. Stationary environment 2 : R n = O ( ln n ) . 2 T.L. Lai and H. Robbins: “Asymptotically efficient adaptive allocation rules” Adv. in Appl.Math. , 6:4-22, 1985. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  7. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  8. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  9. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  10. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  11. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  12. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  13. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results The Exp3 Algorithm Parameters: η – learning rate, γ – exploration rate. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) select I t ∈ { 1 , . . . , N } randomly according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) Compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N : g ′ t ( i , Y t ) = I ( I t = i ) g ( I t , Y t ) / p i , t . (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  14. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Interpretation Fully observable case; g ( i , Y t ) is known for each round. Exponentially weighted predictors: 3 n � w i , n = exp ( η g ( i , Y t )) , γ = 0 t = 1 Here g ′ t ( i , Y t ) is an unbiased estimate of the reward g ( i , Y t ) , n n � � g t ( i , Y t ) ∼ g ′ t ( i , Y t ) . t = 1 t = 1 3 Weighted majority – Littlestone & Warmuth (1994), Aggregating strategies – Vovk (1990), Freund & Schapire (1997,1999). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  15. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Interpretation Fully observable case; g ( i , Y t ) is known for each round. Exponentially weighted predictors: 3 n � w i , n = exp ( η g ( i , Y t )) , γ = 0 t = 1 Here g ′ t ( i , Y t ) is an unbiased estimate of the reward g ( i , Y t ) , n n � � g t ( i , Y t ) ∼ g ′ t ( i , Y t ) . t = 1 t = 1 3 Weighted majority – Littlestone & Warmuth (1994), Aggregating strategies – Vovk (1990), Freund & Schapire (1997,1999). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  16. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Interpretation Fully observable case; g ( i , Y t ) is known for each round. Exponentially weighted predictors: 3 n � w i , n = exp ( η g ( i , Y t )) , γ = 0 t = 1 Here g ′ t ( i , Y t ) is an unbiased estimate of the reward g ( i , Y t ) , n n � � g t ( i , Y t ) ∼ g ′ t ( i , Y t ) . t = 1 t = 1 3 Weighted majority – Littlestone & Warmuth (1994), Aggregating strategies – Vovk (1990), Freund & Schapire (1997,1999). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  17. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Exp3: Unbiasedness of Payoff Estimates E [ g ′ t ( i , Y t ) | F t ] = � N j = 1 E [ I ( I t = i ) g ( I t , Y t ) / p i , t | F t ] P ( I t = j | F t ) = g ( i , Y t ) / p i , t P ( I t = i | F t ) = g ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  18. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Exp3: Unbiasedness of Payoff Estimates E [ g ′ t ( i , Y t ) | F t ] = � N j = 1 E [ I ( I t = i ) g ( I t , Y t ) / p i , t | F t ] P ( I t = j | F t ) = g ( i , Y t ) / p i , t P ( I t = i | F t ) = g ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  19. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Exp3: Unbiasedness of Payoff Estimates E [ g ′ t ( i , Y t ) | F t ] = � N j = 1 E [ I ( I t = i ) g ( I t , Y t ) / p i , t | F t ] P ( I t = j | F t ) = g ( i , Y t ) / p i , t P ( I t = i | F t ) = g ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  20. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Exp3: Unbiasedness of Payoff Estimates E [ g ′ t ( i , Y t ) | F t ] = � N j = 1 E [ I ( I t = i ) g ( I t , Y t ) / p i , t | F t ] P ( I t = j | F t ) = g ( i , Y t ) / p i , t P ( I t = i | F t ) = g ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  21. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  22. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  23. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  24. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  25. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  26. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  27. Improved Bandit Algorithms Universal Prediction with Expert Advice Some Previous Results Example: Dynamic Pricing, single product Player: A vendor – wants to sell a product Adversary: A new customer in each round Problem: Select the “right” price (high enough to make profit, low enough to cut a deal!) Expert i suggests price p 1 , i Highest price the customer is willing to accept: p 2 (unknown! never revealed!) Payoff: g i = p 1 , i I ( p 1 , i ≤ p 2 ) ; vendor learns only g I t ! y = ( p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  28. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  29. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Some issues Practical performance is often very poor We would like to use bandit algorithms in Poker for opponent modelling! Bound scales with N like O ( N ln N ) Fully observable case: O ( ln N ) Possible remedy(?): Allow best expert change with time 4 More information/structure is often available – why not exploiting it? 4 M.K. Warmuth, M. Hebster: “Tracking the best expert”, Machine Learning , 32:151–178, 1998. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  30. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Some issues Practical performance is often very poor We would like to use bandit algorithms in Poker for opponent modelling! Bound scales with N like O ( N ln N ) Fully observable case: O ( ln N ) Possible remedy(?): Allow best expert change with time 4 More information/structure is often available – why not exploiting it? 4 M.K. Warmuth, M. Hebster: “Tracking the best expert”, Machine Learning , 32:151–178, 1998. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  31. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Some issues Practical performance is often very poor We would like to use bandit algorithms in Poker for opponent modelling! Bound scales with N like O ( N ln N ) Fully observable case: O ( ln N ) Possible remedy(?): Allow best expert change with time 4 More information/structure is often available – why not exploiting it? 4 M.K. Warmuth, M. Hebster: “Tracking the best expert”, Machine Learning , 32:151–178, 1998. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  32. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Some issues Practical performance is often very poor We would like to use bandit algorithms in Poker for opponent modelling! Bound scales with N like O ( N ln N ) Fully observable case: O ( ln N ) Possible remedy(?): Allow best expert change with time 4 More information/structure is often available – why not exploiting it? 4 M.K. Warmuth, M. Hebster: “Tracking the best expert”, Machine Learning , 32:151–178, 1998. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  33. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Some issues Practical performance is often very poor We would like to use bandit algorithms in Poker for opponent modelling! Bound scales with N like O ( N ln N ) Fully observable case: O ( ln N ) Possible remedy(?): Allow best expert change with time 4 More information/structure is often available – why not exploiting it? 4 M.K. Warmuth, M. Hebster: “Tracking the best expert”, Machine Learning , 32:151–178, 1998. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  34. Improved Bandit Algorithms Universal Prediction with Expert Advice Issues Some issues Practical performance is often very poor We would like to use bandit algorithms in Poker for opponent modelling! Bound scales with N like O ( N ln N ) Fully observable case: O ( ln N ) Possible remedy(?): Allow best expert change with time 4 More information/structure is often available – why not exploiting it? 4 M.K. Warmuth, M. Hebster: “Tracking the best expert”, Machine Learning , 32:151–178, 1998. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  35. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  36. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Side information Bandit Problems with Side Information Player receives information C t about the environment state 1 Player selects expert I t 2 Expert I t plays against the advisory, knowing C t 3 Player receives payoff g ( I t , Y t ) 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  37. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Side information Bandit Problems with Side Information Player receives information C t about the environment state 1 Player selects expert I t 2 Expert I t plays against the advisory, knowing C t 3 Player receives payoff g ( I t , Y t ) 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  38. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Side information Bandit Problems with Side Information Player receives information C t about the environment state 1 Player selects expert I t 2 Expert I t plays against the advisory, knowing C t 3 Player receives payoff g ( I t , Y t ) 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  39. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Side information Bandit Problems with Side Information Player receives information C t about the environment state 1 Player selects expert I t 2 Expert I t plays against the advisory, knowing C t 3 Player receives payoff g ( I t , Y t ) 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  40. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 The Key Observation Hypothesis In Exp3 any unbiased estimate of the immediate payoffs will do.. Estimates with smaller variance should lead to more efficient algorithms. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  41. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 The Key Observation Hypothesis In Exp3 any unbiased estimate of the immediate payoffs will do.. Estimates with smaller variance should lead to more efficient algorithms. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  42. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 The Key Observation Hypothesis In Exp3 any unbiased estimate of the immediate payoffs will do.. Estimates with smaller variance should lead to more efficient algorithms. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  43. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Example: Dynamic Pricing, multiple products Side information: the cost of the product v Payoff: p 1 I ( p 1 ≤ p 2 ) + ( 1 − α ) v I ( p 1 < p 2 ) . y = ( v , p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) + ( 1 − α ) v I ( p 1 , i > p 2 ) . Hypothesis: One should be able to reduce payoff variance given the knowledge of v L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  44. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Example: Dynamic Pricing, multiple products Side information: the cost of the product v Payoff: p 1 I ( p 1 ≤ p 2 ) + ( 1 − α ) v I ( p 1 < p 2 ) . y = ( v , p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) + ( 1 − α ) v I ( p 1 , i > p 2 ) . Hypothesis: One should be able to reduce payoff variance given the knowledge of v L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  45. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Example: Dynamic Pricing, multiple products Side information: the cost of the product v Payoff: p 1 I ( p 1 ≤ p 2 ) + ( 1 − α ) v I ( p 1 < p 2 ) . y = ( v , p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) + ( 1 − α ) v I ( p 1 , i > p 2 ) . Hypothesis: One should be able to reduce payoff variance given the knowledge of v L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  46. Improved Bandit Algorithms Improved Payoff-Estimation Generalized Exp3 Example: Dynamic Pricing, multiple products Side information: the cost of the product v Payoff: p 1 I ( p 1 ≤ p 2 ) + ( 1 − α ) v I ( p 1 < p 2 ) . y = ( v , p 2 , p 1 , 1 , . . . , p 1 , N ) , so g ( i , y ) = p 1 , i I ( p 1 , i ≤ p 2 ) + ( 1 − α ) v I ( p 1 , i > p 2 ) . Hypothesis: One should be able to reduce payoff variance given the knowledge of v L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  47. Improved Bandit Algorithms Improved Payoff-Estimation Theoretical Results Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  48. Improved Bandit Algorithms Improved Payoff-Estimation Theoretical Results The Exp3G Algorithm Parameters: real numbers 0 < η, γ < 1. Initialization: w 0 = ( 1 , . . . , 1 ) T ; For each round t = 1 , 2 , . . . (1) observe C t , select I t ∈ { 1 , . . . , N } according to w i , t − 1 + γ p i , t = ( 1 − γ ) N ; � N k = 1 w k , t − 1 (2) observe g t = g ( I t , Y t ) ; (3) based on g t , C t , compute the feedbacks g ′ t ( i , Y t ) , i = 1 , . . . , N ; (4) compute w i , t = w i , t − 1 e η g ′ t ( i , Y t ) . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  49. Improved Bandit Algorithms Improved Payoff-Estimation Theoretical Results Exp3G: Expected Regret Assumptions: E [ g ( i , Y t ) | C t , I t − 1 , Y t − 1 ] ( A 1 ) ≤ 1 ; t ( i , Y t ) | C t , I t − 1 , Y t − 1 ] = E [ g ( i , Y t ) | C t , I t − 1 , Y t − 1 ]; ( A 2 ) E [ g ′ ≤ σ 2 ; t ( i , Y t ) | C t , I t − 1 , Y t − 1 ] Var [ g ′ ( A 3 ) ( A 4 ) | g ′ t ( i , Y t ) | ≤ B . Theorem Consider algorithm Exp3G. Assume A1–A4. Then for γ = 0 and a suitable η = η n , n sufficiently large, n n � � � E [ g ( i , Y t )] − E [ g ( I t , Y t )] ≤ ( 1 + σ 2 ) n ln N . max i t = 1 t = 1 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  50. Improved Bandit Algorithms Improved Payoff-Estimation Theoretical Results Exp3G: PAC-bounds on Regret Theorem Consider algorithm Exp3G. Assume A1–A4 and further, assume that | g ( i , Y t ) | ≤ 1. Then, for any δ > 0, for a suitable η = η n with n sufficiently large, the following bound on the regret of Exp3G holds with probability at least 1 − δ : � (( 1 + σ 2 ) ln N ) 1 / 2 + G in − ˆ G n ≤ n 1 / 2 max i � 1 / 2 � √ � N + 1 + 2 ( B + 1 ) � N + 1 � ( 2 + 2 σ ) ln ln . δ 3 δ L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  51. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  52. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation Likelihood-ratio-Based Payoff Estimation Assumptions Explicit model of the game played (actions). Experts have probabilistic action selection strategies. Probability of any action (action-sequence) can be queried for any expert. Payoff-estimate (single-stage games): t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . This estimate is unbiased. Extension to multi-stage games is straightforward (use the chain-rule to show unbiasedness). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  53. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation Likelihood-ratio-Based Payoff Estimation Assumptions Explicit model of the game played (actions). Experts have probabilistic action selection strategies. Probability of any action (action-sequence) can be queried for any expert. Payoff-estimate (single-stage games): t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . This estimate is unbiased. Extension to multi-stage games is straightforward (use the chain-rule to show unbiasedness). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  54. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation Likelihood-ratio-Based Payoff Estimation Assumptions Explicit model of the game played (actions). Experts have probabilistic action selection strategies. Probability of any action (action-sequence) can be queried for any expert. Payoff-estimate (single-stage games): t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . This estimate is unbiased. Extension to multi-stage games is straightforward (use the chain-rule to show unbiasedness). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  55. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation Likelihood-ratio-Based Payoff Estimation Assumptions Explicit model of the game played (actions). Experts have probabilistic action selection strategies. Probability of any action (action-sequence) can be queried for any expert. Payoff-estimate (single-stage games): t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . This estimate is unbiased. Extension to multi-stage games is straightforward (use the chain-rule to show unbiasedness). L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  56. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation LExp: LR-based Payoff Estimation # 1 Simple LR-based estimate: t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . Insight Large likehood ratios are likely to yield large variance. Idea: When π i ( A t | C t ) /π I t ( A t | C t ) is big, make it equal to 0 and compensate for the bias introduced. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  57. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation LExp: LR-based Payoff Estimation # 2 Simple LR-based estimate: t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . Let φ t ( I t , A t , i ) be such that φ t ( I t , A t , i ) = 1 denotes an event when the LRs are big: � π i ( a | C t ) π k ( a | C t ) > p k , t � φ t ( k , a , i ) = I . p i , t Modified LR-based estimate: ( 1 − φ t ( I t , A t , i )) π i ( A t | C t ) g ′ t ( i , Y t ) = π I t ( A t | C t ) g ( I t , Y t ) . � N j = 1 p j , t ( 1 − φ t ( j , A t , i )) L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  58. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation LExp: LR-based Payoff Estimation # 2 Simple LR-based estimate: t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . Let φ t ( I t , A t , i ) be such that φ t ( I t , A t , i ) = 1 denotes an event when the LRs are big: � π i ( a | C t ) π k ( a | C t ) > p k , t � φ t ( k , a , i ) = I . p i , t Modified LR-based estimate: ( 1 − φ t ( I t , A t , i )) π i ( A t | C t ) g ′ t ( i , Y t ) = π I t ( A t | C t ) g ( I t , Y t ) . � N j = 1 p j , t ( 1 − φ t ( j , A t , i )) L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  59. Improved Bandit Algorithms Improved Payoff-Estimation Likelihood-ratio Based Payoff Estimation LExp: LR-based Payoff Estimation # 2 Simple LR-based estimate: t ( i , Y t ) = π i ( A t | C t ) g ′ π I t ( A t | C t ) g ( I t , Y t ) . Let φ t ( I t , A t , i ) be such that φ t ( I t , A t , i ) = 1 denotes an event when the LRs are big: � π i ( a | C t ) π k ( a | C t ) > p k , t � φ t ( k , a , i ) = I . p i , t Modified LR-based estimate: ( 1 − φ t ( I t , A t , i )) π i ( A t | C t ) g ′ t ( i , Y t ) = π I t ( A t | C t ) g ( I t , Y t ) . � N j = 1 p j , t ( 1 − φ t ( j , A t , i )) L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  60. Improved Bandit Algorithms Improved Payoff-Estimation Control-variates Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  61. Improved Bandit Algorithms Improved Payoff-Estimation Control-variates CExp3: Control-variates # 1 Motivation: In dynamic pricing, the product C t controls to a large extent the distribution of the actual payoffs g ( i , Y t ) – hence also the variance. Idea: consider the payoffs compensated for C t instead of the original payoffs: g c ( i , Y t ) = g ( i , Y t ) − r ( C t ) , g ′ g ′ c , t ( i , Y t ) = t ( i , Y t ) − r ( C t ) . Here r ( C t ) is the mean payoff when seeing C t . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  62. Improved Bandit Algorithms Improved Payoff-Estimation Control-variates CExp3: Control-variates # 1 Motivation: In dynamic pricing, the product C t controls to a large extent the distribution of the actual payoffs g ( i , Y t ) – hence also the variance. Idea: consider the payoffs compensated for C t instead of the original payoffs: g c ( i , Y t ) = g ( i , Y t ) − r ( C t ) , g ′ g ′ c , t ( i , Y t ) = t ( i , Y t ) − r ( C t ) . Here r ( C t ) is the mean payoff when seeing C t . L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  63. Improved Bandit Algorithms Improved Payoff-Estimation Control-variates CExp3: Control-variates # 2 Effect: t ( i , Y t ) | Y t − 1 , I t − 1 ] is reduced. Previous analysis can Var [ g ′ be repeated to show that this is beneficial – regret bounds for compressed range payoffs carry over to the regret defined with the unmodified payoffs. Intuitive explanation: Compresses the range of payoffs. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  64. Improved Bandit Algorithms Improved Payoff-Estimation Control-variates CExp3: Control-variates # 2 Effect: t ( i , Y t ) | Y t − 1 , I t − 1 ] is reduced. Previous analysis can Var [ g ′ be repeated to show that this is beneficial – regret bounds for compressed range payoffs carry over to the regret defined with the unmodified payoffs. Intuitive explanation: Compresses the range of payoffs. L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  65. Improved Bandit Algorithms Experimental Results Dynamic Pricing Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  66. Improved Bandit Algorithms Experimental Results Dynamic Pricing Dynamic pricing Experts: E4 E5 Customers: p 2 = 1 . 1 v + β − 50 E3 500 , E2 E1 where β ∼ B ( 100 , 0 . 5 ) . 0 . 9 v 1 . 1 v v v + 0 . 02 b L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  67. Improved Bandit Algorithms Experimental Results Dynamic Pricing Results: Almost deterministic experts b = 0.05 0.014 EXP3 CEXP3 0.012 LEXP 0.01 average regret 0.008 0.006 0.006 0.005 0.004 0.003 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 iteration L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  68. Improved Bandit Algorithms Experimental Results Dynamic Pricing Dynamic pricing: Statistics # 1 b = 0 . 05 E [ g ′ t ( i , Y t )] i=1 i=2 i=3 i=4 i=5 Exp3 0.388 0.399 0.401 0.371 0.396 CExp3 0.390 0.399 0.398 0.371 0.398 LExp 0.390 0.402 0.400 0.368 0.399 E [ g ( i , Y t )] 0.390 0.400 0.399 0.371 0.399 � Var [ g ′ t ( i , Y t )] i=1 i=2 i=3 i=4 i=5 Exp3 1.782 1.435 1.427 2.097 1.573 CExp3 0.467 0.476 0.473 0.332 0.472 LExp 0.739 0.788 0.500 1.671 0.688 � Var [ g ( i , Y t )] 0.143 0.148 0.145 0.129 0.144 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  69. Improved Bandit Algorithms Experimental Results Dynamic Pricing Results: Heavily randomized experts b = 0.3 0.03 EXP3 CEXP3 0.025 LEXP 0.02 0.017 average regret 0.014 0.012 0.01 0.008 0.006 0.005 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 iteration L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  70. Improved Bandit Algorithms Experimental Results Dynamic Pricing Dynamic pricing: Statistics # 2 b = 0 . 3 E [ g ′ t ( i , Y t )] i=1 i=2 i=3 i=4 i=5 Exp3 0.338 0.351 0.347 0.385 0.383 CExp3 0.343 0.354 0.348 0.381 0.382 LExp 0.343 0.356 0.351 0.383 0.384 E [ g ( i , Y t )] 0.343 0.356 0.350 0.383 0.384 � Var [ g ′ t ( i ; Y t )] i=1 i=2 i=3 i=4 i=5 Exp3 2.107 1.929 2.014 1.046 1.169 CExp3 0.735 0.724 0.726 0.744 0.745 LExp 0.856 0.573 0.651 0.475 0.412 � Var [ g ( i , Y t )] 0.153 0.151 0.150 0.141 0.143 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  71. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Outline Universal Prediction with Expert Advice 1 Prediction with Expert Advice Some Previous Results Issues Improved Payoff-Estimation 2 Generalized Exp3 Theoretical Results Likelihood-ratio Based Payoff Estimation Control-variates Experimental Results 3 Dynamic Pricing Dynamic Pricing – Tracking Experiments Experiments with Poker Conclusions 4 L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  72. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Experiments with Tracking Algorithms Goal: minimize regret over the sequence of best experts where the frequency of expert changes is upper bounded Problem: Exp3 and variants can run into problems (weights converging too fast – too slow response at changepoints Warmuth & Herbster: Tracking the Best Expert (ML, 1998); Fixed-Share Algorithm: � N j = 1 w j , t − 1 e η g ′ t ( j , Y t ) + ( 1 − α ) w i , t − 1 e η g ′ t ( i , Y t ) . w i , t = α N L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  73. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Experiments with Tracking Algorithms Goal: minimize regret over the sequence of best experts where the frequency of expert changes is upper bounded Problem: Exp3 and variants can run into problems (weights converging too fast – too slow response at changepoints Warmuth & Herbster: Tracking the Best Expert (ML, 1998); Fixed-Share Algorithm: � N j = 1 w j , t − 1 e η g ′ t ( j , Y t ) + ( 1 − α ) w i , t − 1 e η g ′ t ( i , Y t ) . w i , t = α N L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  74. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Experiments with Tracking Algorithms Goal: minimize regret over the sequence of best experts where the frequency of expert changes is upper bounded Problem: Exp3 and variants can run into problems (weights converging too fast – too slow response at changepoints Warmuth & Herbster: Tracking the Best Expert (ML, 1998); Fixed-Share Algorithm: � N j = 1 w j , t − 1 e η g ′ t ( j , Y t ) + ( 1 − α ) w i , t − 1 e η g ′ t ( i , Y t ) . w i , t = α N L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  75. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Experiments with Tracking Algorithms Goal: minimize regret over the sequence of best experts where the frequency of expert changes is upper bounded Problem: Exp3 and variants can run into problems (weights converging too fast – too slow response at changepoints Warmuth & Herbster: Tracking the Best Expert (ML, 1998); Fixed-Share Algorithm: � N j = 1 w j , t − 1 e η g ′ t ( j , Y t ) + ( 1 − α ) w i , t − 1 e η g ′ t ( i , Y t ) . w i , t = α N L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  76. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Experiments with Tracking Algorithms Goal: minimize regret over the sequence of best experts where the frequency of expert changes is upper bounded Problem: Exp3 and variants can run into problems (weights converging too fast – too slow response at changepoints Warmuth & Herbster: Tracking the Best Expert (ML, 1998); Fixed-Share Algorithm: � N j = 1 w j , t − 1 e η g ′ t ( j , Y t ) + ( 1 − α ) w i , t − 1 e η g ′ t ( i , Y t ) . w i , t = α N L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  77. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Parameters of Experiments E [ g ( 1 , Y t )] E [ g ( 2 , Y t )] E [ g ( 3 , Y t )] E [ g ( 4 , Y t )] E [ g ( 5 , Y t )] round m k 0 100 1.1 0.3906 0.4000 0.3992 0.3714 0.3989 5000 10 0.1 0.3767 0.3641 0.3716 0.371 0.3760 10000 20 0.9 0.3607 0.3604 0.3605 0.3615 0.3606 15000 100 1.0 0.3785 0.3766 0.3806 0.3687 0.3822 b = 0 . 05 p 2 = kv + β − m / 2 , β ∼ B ( m , 0 . 5 ) m L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  78. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments No exploration, fixed-share update-rule 90 fixed-share EXP3(gamma=0) fixed-share CEXP3(gamma=0) fixed-share LEXP(gamma=0) 80 70 cumulative regret 60 50 40 30 20 10 0 0 5000 10000 15000 iteration L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  79. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Results for CExp3 variants 60 CEXP3 restart CEXP3 fixed-share CEXP3 fixed-share CEXP3(gamma=0) 50 cumulative regret 40 30 20 10 0 0 5000 10000 15000 iteration L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

  80. Improved Bandit Algorithms Experimental Results Dynamic Pricing – Tracking Experiments Expert-selection Frequencies (CExp3) CEXP3 restart CEXP3 0.7 0.4 expert 1 expert 1 expert 2 expert 2 expert 3 expert 3 0.35 0.6 expert 4 expert 4 expert 5 expert 5 0.3 0.5 choice probability choice probability 0.25 0.4 0.2 0.3 0.15 0.2 0.1 0.1 0.05 0 0 0 5000 10000 15000 0 5000 10000 15000 iteration iteration fixed-share CEXP3(gamma=0) fixed-share CEXP3 0.4 0.4 expert 1 expert 1 expert 2 expert 2 expert 3 expert 3 0.35 0.35 expert 4 expert 4 expert 5 expert 5 0.3 0.3 choice probability choice probability 0.25 0.25 0.2 0.2 0.15 0.15 0.1 0.1 0.05 0.05 0 0 0 5000 10000 15000 0 5000 10000 15000 iteration iteration L. Kocsis and Cs. Szepesv´ ari Improved Bandit Algorithms

Recommend

Estimation theory Parametric estimation Properties of estimators Minimum variance

Estimation theory Parametric estimation Properties of estimators Minimum variance

Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let X be a scalar

1.11k views • 44 slides
Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science

Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science

Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science and Technology wzhuai@ust.hk April 3, 2019 Robust Estimation and Generative Adversarial Nets [GLYZ18] Generative Adversarial Nets for Robust

252 views • 24 slides
Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1

Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1

Variance of an Indicator Mathematics for Computer Science MIT 6.042J/18.062J IanindicatorwithE[I]=p: Var[I]:: = E[(I p) 2 ] Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1 variance.2

104 views • 6 slides
Analysis of variance and regression December 4, 2007 Variance component models Variance

Analysis of variance and regression December 4, 2007 Variance component models Variance

Analysis of variance and regression December 4, 2007 Variance component models Variance components One-way anova with random variation estimation interpretations Two-way anova with random variation Crossed random effects

803 views • 78 slides
State Estimation and Contingency Analysis of the Power Grid in a Cyber-Adversarial Environment

State Estimation and Contingency Analysis of the Power Grid in a Cyber-Adversarial Environment

State Estimation and Contingency Analysis of the Power Grid in a Cyber-Adversarial Environment Robin Berthier 1 , Rakesh Bobba 1 , Matt Davis 2 , Kate Rogers 2 , and Saman Zonouz 3 1 Information Trust Institute 3 Department of Electrical 2

905 views • 47 slides
Variance Reduction Methods for Parametric Bootstrap MSE-Estimation Session: Different Inferential

Variance Reduction Methods for Parametric Bootstrap MSE-Estimation Session: Different Inferential

Variance Reduction Methods for Parametric Bootstrap MSE-Estimation Session: Different Inferential Issues in Area Level Models Jan Pablo Burgard Wirtschafts- und Sozialstatistik Universitt Trier, FB IV, VWL 04.09.2013 SAE Bangkok, 04.09.2013

446 views • 29 slides
Control Regularization for Reduced Variance Reinforcement Learning Richard Cheng, Abhinav Verma,

Control Regularization for Reduced Variance Reinforcement Learning Richard Cheng, Abhinav Verma,

Control Regularization for Reduced Variance Reinforcement Learning Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri Yisong Yue, Joel W. Burdick Reinforcement Learning Reinforcement learning (RL) studies how to use data from

204 views • 9 slides
Variance Estimation for Networking Congestion Avoidance Algorithm. Olga I. Bogoiavlenskaia

Variance Estimation for Networking Congestion Avoidance Algorithm. Olga I. Bogoiavlenskaia

Variance Estimation for Networking Congestion Avoidance Algorithm. Olga I. Bogoiavlenskaia PetrSU, Department of Computer Science olbgvl@cs.karelia.ru TCP Congestion Control The paradigm of Distributed Control in Packet Switching Network

175 views • 17 slides
Adversarial Contrastive Estimation ACL 2018 *AVISHEK (JOEY) BOSE, *HUAN LING, *YANSHUAI CAO

Adversarial Contrastive Estimation ACL 2018 *AVISHEK (JOEY) BOSE, *HUAN LING, *YANSHUAI CAO

Adversarial Contrastive Estimation Adversarial Contrastive Estimation ACL 2018 *AVISHEK (JOEY) BOSE, *HUAN LING, *YANSHUAI CAO Avishek (Joey) Bose* , Huan Ling*, Yanshuai Cao* | Borealis AI, University of Toronto | August 2, 2018 1 / 29

689 views • 29 slides
Principal Component of Explained Variance High-Dimensional Estimation and Inference Max Turgeon

Principal Component of Explained Variance High-Dimensional Estimation and Inference Max Turgeon

Principal Component of Explained Variance High-Dimensional Estimation and Inference Max Turgeon February 14th, 2020 University of Manitoba Departments of Statistics and Computer Science 1/47 Introduction In modern statistics, we often

844 views • 55 slides
Variance Estimation for Survey-Weighted Data Using Bootstrap Resampling Methods: 2013

Variance Estimation for Survey-Weighted Data Using Bootstrap Resampling Methods: 2013

Variance Estimation for Survey-Weighted Data Using Bootstrap Resampling Methods: 2013 Methods-of-Payment Survey Questionnaire Heng Chen and Q. Rallye Shen 2017 STATA The views expressed in this presentation are those of the author. No

249 views • 10 slides
Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov Julien Mairal andrei.kulunchakov@inria.fr julien.mairal.@inria.fr International Conference on Machine Learning, 2019 Poster event-4062, (Jun 12th,

317 views • 6 slides
Introduction to Generative Adversarial Network (GAN) Hongsheng Li Department of Electronic

Introduction to Generative Adversarial Network (GAN) Hongsheng Li Department of Electronic

Introduction to Generative Adversarial Network (GAN) Hongsheng Li Department of Electronic Engineering Chinese University of Hong Kong Adversarial adj. 1 Generative Models Density Estimation ( | ) p y x

1.35k views • 98 slides
Modelling and Estimation of Stochastic Dependence Uwe Schmock Based on joint work with Dr.

Modelling and Estimation of Stochastic Dependence Uwe Schmock Based on joint work with Dr.

Definitions of dependence measures and basic properties Asymptotic variance of the tau-estimators for copulas Asymptotic variance for elliptical distributions Modelling and Estimation of Stochastic Dependence Uwe Schmock Based on joint work

486 views • 46 slides
Reduced-order models for uncertainty quantification and parameter estimation in cardiac models

Reduced-order models for uncertainty quantification and parameter estimation in cardiac models

Reduced-order models for uncertainty quantification and parameter estimation in cardiac models Stefano Pagani Alfio Quarteroni Andrea Manzoni MOX-Dipartimento di

174 views • 15 slides
Variance Estimation in Presence of Imputation: an Application to ISTAT Survey Data Marco Di Zio,

Variance Estimation in Presence of Imputation: an Application to ISTAT Survey Data Marco Di Zio,

Variance Estimation in Presence of Imputation: an Application to ISTAT Survey Data Marco Di Zio, Stefano Falorsi, Ugo Guarnera, Orietta Luzi, Paolo Righi paolo.righi@istat.it European Conference on Quality in Official Statistics - 2008, Rome 8-11

731 views • 18 slides
Reinforcement Learning n-armed bandit Kevin Spiteri April 21, 2015 n-armed bandit n-armed

Reinforcement Learning n-armed bandit Kevin Spiteri April 21, 2015 n-armed bandit n-armed

Reinforcement Learning n-armed bandit Kevin Spiteri April 21, 2015 n-armed bandit n-armed bandit 0.9 0.5 0.1 0.9 0.5 0.1 0.0 0.0 0.0 0.0 estimate n-armed bandit n-armed bandit 0.9 0.5 0.1 0.9 0.5 0.1 0 0.0 0.0 0.0 0.0

676 views • 21 slides
Performance Estimation and Regularization Kasthuri Kannan, PhD. Machine

Performance Estimation and Regularization Kasthuri Kannan, PhD. Machine

Performance Estimation and Regularization Kasthuri Kannan, PhD. Machine Learning, Spring 2018 Bias-Variance Tradeoff Fundamental to machine learning approaches Bias-Variance

766 views • 48 slides
One Armed Bandit source: http://dogbeforewicket.blogspot.ca EECS 1030 moodle.yorku.ca One Armed

One Armed Bandit source: http://dogbeforewicket.blogspot.ca EECS 1030 moodle.yorku.ca One Armed

One Armed Bandit source: http://dogbeforewicket.blogspot.ca EECS 1030 moodle.yorku.ca One Armed Bandit Utility /** * Returns the winnings from one pull of the one armed * bandit. * * @param coin the coin deposited in the one armed bandit.

623 views • 58 slides
Estimation: Sample Complexity and the Bias-Variance Tradeoff CMPUT 296: Basics of Machine

Estimation: Sample Complexity and the Bias-Variance Tradeoff CMPUT 296: Basics of Machine

Estimation: Sample Complexity and the Bias-Variance Tradeoff CMPUT 296: Basics of Machine Learning Textbook 3.4-3.5 Logistics Reminders: Thought Question 1 (due Thursday, September 17 ) Assignment 1 (due Thursday, September 24 )

983 views • 16 slides
Optimal state estimation for numerical weather prediction using reduced order models A.S. Lawless

Optimal state estimation for numerical weather prediction using reduced order models A.S. Lawless

4D-Var Model reduction in DA Unstable systems Numerical results Conclusions Optimal state estimation for numerical weather prediction using reduced order models A.S. Lawless 1 , C. Boess 1 , N.K. Nichols 1 and A. Bunse-Gerstner 2 1 School of

737 views • 51 slides
Interval Estimation Edwin Leuven Interval estimation While an estimator may be unbiased or

Interval Estimation Edwin Leuven Interval estimation While an estimator may be unbiased or

Interval Estimation Edwin Leuven Interval estimation While an estimator may be unbiased or consistent, a given estimate will never equal the true value this point estimate does not give a sense of closeness, while the variance

898 views • 36 slides
Announcements HW 1 is due now 1 CS6501: T opics in Learning and Game Theory (Fall 2019)

Announcements HW 1 is due now 1 CS6501: T opics in Learning and Game Theory (Fall 2019)

Announcements HW 1 is due now 1 CS6501: T opics in Learning and Game Theory (Fall 2019) Adversarial Multi-Armed Bandits Instructor: Haifeng Xu Outline The Adversarial Multi-armed Bandit Problem A Basic Algorithm: Exp3 Regret

864 views • 53 slides
Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji 1 , Zhe Wang 1 , Yi Zhou 2 , Yingbin Liang 1 1 Ohio State University, 2 Duke University ICML 2019 K. Ji, Z. Wang, Y. Zhou, Y. Liang Zeroth-Order

327 views • 9 slides

More recommend