Advanced Machine Learning CS 7140 - Spring 2019 Lecture 24: - PowerPoint PPT Presentation

Advanced Machine Learning CS 7140 - Spring 2019 Lecture 24: Bayesian Optimization Jan-Willem van de Meent Slide credits: Ryan Adams, Nando de Freitas

Background: Multi-Armed Bandits • Problem: Which machine has highest rate of payout? • Trade-off: Exploration (trying a new machine) vs 
 Exploitation (playing machine with best returns so far) • Regret: Difference between reward of action, and reward 
 of optimal action (with benefit of hindsight)

Background: Multi-Armed Bandits • Problem: Which machine has highest rate of payout? • Trade-off: Exploration (trying a new machine) vs 
 Exploitation (playing machine with best returns so far) • Regret: Difference between reward of action, and reward 
 of optimal action (with benefit of hindsight)

Example: Thompson Sampling Goal: Use A/B testing to optimize button click rate Thompson Sampling Bandit Require : � ; � : hyperparameters of the beta prior 1: Initialize n a ; 0 ¼ n a ; 1 ¼ i ¼ 0 for all a 2: repeat for a ¼ 1 ; . . . ; K do 3: w a � beta ð � þ n a ; 1 ; � þ n a ; 0 Þ 4: ~ 5: end for a i ¼ arg max a ~ 6: w a 7: Observe y i by pulling arm a i if y i ¼ 0 then 8: n a i ; 0 ¼ n a i ; 0 þ 1 9: 10: else n a i ; 1 ¼ n a i ; 1 þ 1 11: 12: end if i ¼ i þ 1 13: 14: until stopping criterion reached

Bayesian Optimization 3 2 current ! 1 best 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization 3 2 current ! 1 best 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization 3 2 current ! 1 best 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization 3 2 current ! 1 best 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization 3 2 current ! 1 best 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Goal: Optimize unknown cost function 
 (continuous version of bandit problem)

Bayesian Optimization 3 2 current ! 1 best 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Problem: Which point should we evaluate next?

Bayesian Optimization 3 2 1 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Idea 1: Model uncertainty about objective function

Bayesian Optimization 3 2 1 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Idea 2: Define acquisition function 
 that balances exploration and exploitation

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Bayesian Optimization

Intuition: Why does Bayes Opt work? Idea: Use confidence bounds to adaptively eliminate regions in search space that are not likely to contain optimum

Modeling Uncertainty

<latexit sha1_base64="fVB4z3/k8AXCtxB7osjfSKqlHG4=">AGFnicfZTNbtQwEIDd0oUS/lo4conYS4tWVdJWbS9IVEFx1L1T2qWleNMdq0mTmo7/bP8HjwCD4G4IW6IG/A2ONlIbOIR4pGM94Zjxjh3lChfS8P3Pz9xZ69x8sPnQePX7y9NnS8vMTkRWcwDHJkoyfhVhAQhkcSyoTOMs54DRM4DS8eFvaT6+AC5qxI3mbwzDFY0ZjSrA0qtHSXnClbvTH1+4bN8B8nFI2UpVKu/FKJay6TnB5WeDIiY0kaOrkK7EbpDRyjf1Wr46W+t6aVy3XFvxa6KN6HYyWF74EUaKFJgkCRbi3PdyOVSYS0oS0E5QCMgxucBjODciwymIoaqK1W7DeuQPVZwxCYw03BRORYrlxFKWsGhqycQEBt4MWyuHqtwlAkHrOkVptpxghic/BVZioKkwK0Ony3p5U32NoY+OvbuoVwiGrC3/EG5msDYw7AamRnc+Bv7dhMXvA8gX+QV2JlNhwYXJMsTGLTBeB6HNzPgEwUXAoC1FBmKq+r7W24ClqfCq7E8wab7RSQSPB6Yi0sNsZrCy0Go82dNe152FXZhgZyAxB3Zy24aFx1sYbGFDXEL4u0MoTMm5ImGbPqiWfoak5iO2gyw9Q9LrdMzG2OsLVjPunG8wm12MNWZw51OS6zRHn3selzkOXAscx4emuqZwkNKVSqNqubS/K/u9l7O1g+7o5lOU/DNW+tkgSJtVgNs/OnlDCoyZXVtmBjXkTmzauA8xbYH3AJWneO7/9utnCyfqa7635Hzb7u3v1y7eIXqJXaAX5aBvtovfoAB0jgj6jH+gX+t371Pva+9b7PkXn52qfF6ixej/ApkAOwA=</latexit> Predictive Posterior over Functions 3 2 1 0 − 1 − 2 − 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

<latexit sha1_base64="tLc2wN7JTbsGayUlk4RLdDNbju0=">AGA3icfZTNbtQwEIBd6EJZ/lo4clmxl1ZalaRUbY9VUQUnVKr+SfVq5Tizu1ZjJzhO263lI4/AU3BD3IAbPARvg51GYhNHOFI08nzj+bWjLG5CoI/C3fuLnbu3V960H346PGTp8srz07ytJAUjmapPIsIjkTMCxYiqBs0wC4VECp9HFG6c/vQSZs1QcqVkGQ04mgo0ZJcpujZfzUaihzmLe+MezhnvYQXSr9PJTer41V8qa/NSKwNnHLCydpouR+sB+Xq+UJYCX1UrYPRyuIPHKe04CAUTUien4dBpoaSMVoAqaLixwyQi/IBM6tKAiHfKjLzEyvpj0Kh3qcCgWC1sw04TknauptOjiv79KpdQy7rbaHGp3Sgw2U1G3irjpdnEMY1vlMjIdR0kBRh+3TM6Gy9HoQb26aBSIgrItwJBvZrAhMJICpkZ3MQbu34TFbILIF/UOAwF40EAVc05ZyIWONLoObc1geDyAsJLhGNI67oTHGg29Ra1Pqu3heW20xrUAyFoYrM5zCVqoZkH3bSdeNhH9swrKagSEv0qp0mRQtbeGzhQ9KDZDNCaPUJWc6SVHj5jOfock7GvtNkjql67I5M7NWNiXdiNm3Hsynz2MNGZw6NG5d5gkh7lW2fcZqBJCqV7tJdMTVNGcq15Xe+FZM/N/K6pvO9k19KN0/ivS+8UgaJeVg1mvnTyiVcZ1zWbZgE1nHbhvXAmYNsCqwI+17FzZfN1842VgPg/Xw2Z/d696+ZbQC/QSraIQbaNd9A4doGNE0Wf0Hf1CvzufOl86XzvfbtE7C5XNc1RbnZ9/ARGkNDA=</latexit> <latexit sha1_base64="LozS45D0JFnutV5rNRSTfUiIwM=">AGAnicfZRLb9QwEIBd6EJZXi0cuazYA620WiWlanusiqpyLFVfUr2qHGd216rtBNvpy/KNn8Cv4Ia4od7gT/BvSLKR2MQRjpSMPN+8PBNHKWfaBMGfhQcPFzuPHi896T59vzFy+WVyc6yRSFY5rwRJ1FRANnEo4NMxzOUgVERBxOo8sPhf70CpRmiTwytymMBJlINmaUmHzrYnk47mHNRA8buDF2/8CtYpGt4it749YGvcuZNOiVn3draxfL/WAYlKvnC2El9FG1Di5WFu9xnNBMgDSUE63PwyA1I0uUYZSD6+JMQ0roJZnAeS5KIkCPbFmY69W0R+HIjhNpQNKamSVC2Km3mYB6/ouneaBQdXDVpsjW3iJQbOJrFtFwnW7OIZxfshlZjaOeAbOHu7vOhsMNt8PwvUt10AUxBURbgeD/GkCEwUgK2R7YxBubvtMmqmUwz8oKLAiGwUSrmkiBJGxVdA3Xl+PhikzhQUhVgcCdsPnXMePENzm1LfxfPKG2ctriVY9r6J3c5hRaE5dOtBd2+7jzscxuGzRQMacnetNMka2Ezj818SHmQamYIrTEh1Ywn0qtnPEeXczL2g/I5pupx4ZLnf25MPI/ptB1Pp8xjDxudOXTFuMwTRE0EyfuMkxQUMYkqfrprZqacCWa0rfTOt2Ly/1a5vhlsz9WHsnhHkd1zHkjXg5m/ez8CaUqrnNFlS3YRNWxWeNawLQBVgdckPl9FzZvN184WR+GwTD8tNHf2a1uviX0Br1FqyhEW2gHfUQH6BhR9BX9RL/Q786XzrfO986PGfpgobJ5jWqrc/8XaZszVg=</latexit> <latexit sha1_base64="agtxwWGEu/nPnQ5UtKLNHaGVTpw=">AHPXichZRbT9RAFIC7iCvUC6CPvjRuYgA3pEUCPBIMUV8MEm7JzkKm7dntZHtzOoWFyfwzE3+Cz/4A34xvxjd1elncdhqYJs3pOd+5zJzTsWOfJMw0v7Vm7s3ebz+Ym9cfPnr8ZGFx6elxEqXUgSMn8iN6auMEfBLCESPMh9OYAg5sH07s0ZvMfnIBNCFReMiuYugHeBiSAXEwk6rzpZnyIV8GLCegWwYkpAHmFEyFrqRmy74lTAQKr4Gq/o8gtCdMNJOydBj/dyMEhIUAoMx4x8iGog89LJ+V5YgXZaZTsXKTapSNRZnqyu3Je3eFXtURO4aRYJXsphgI2z9UzxXhgvK0Se76aI0aSEifeEvtHdWV8hr+jnix1zcyXoQpWKXS0cu2fL81+Rm7kpAGEzPFxkvQsM2Z9jikjg9CR2kCMXZGeAg9KY4gKTP85kQRsV6aPX5IAoZhE7FjeMgkUV7ijKDk6rW8WRioNW0pbKfbd1zQZ5sWPWyA6HryIWBnM+8Mu7afgqCH7zdFdzsbr7uWutboZQcEvC2ja78qkDQwoQlsj2Rtfa3FaZOKWxD/8hM8OyaiEcOlEQYBlz9AFOKInz0d2MEkpZBvhyA54xJCKHCBSp/crqNp41hwjioF5vNRx6msGyj+R9Wh6bYl0r2KcmDEPG6onjXTOG1gU4VNVYgqEK1XCI05IU6IH4XKfgZTdD4nAzWpP8WUPc5C+vLSc7ESMfa8dgjCntQ68yByMZlmsBUXh2yzyiKgWIW0eynuyTM80lAWMJLu1C9SHi7l7TXk+2J6lBmb9vme0IhHdvPB7N6duqEOtStctkuG7AhrWJF4xrAuAaWB5yR8r6z6rebKhyvr1nmvVxo7OzW958c9pz7YW2rFnalrajvdP2tSPNaX1t/Wr9af1tf2l/b/9o/yzQmVbp80yrPbvf0oKlY=</latexit> Recap: Gaussian Processes Formal View: Prior over Functions Practical View: Generalization of Multivariate Normal