the online approach to machine learning
play

The Online Approach to Machine Learning Nicol` o Cesa-Bianchi - PowerPoint PPT Presentation

Nov 13, 2023 •527 likes •1.16k views

The Online Approach to Machine Learning Nicol` o Cesa-Bianchi Universit` a degli Studi di Milano N. Cesa-Bianchi (UNIMI) Online Approach to ML 1 / 53 Summary My beautiful regret 1 A supposedly fun game Ill play again 2 A graphic

  1. The Online Approach to Machine Learning Nicol` o Cesa-Bianchi Universit` a degli Studi di Milano N. Cesa-Bianchi (UNIMI) Online Approach to ML 1 / 53

  2. Summary My beautiful regret 1 A supposedly fun game I’ll play again 2 A graphic novel 3 The joy of convex 4 The joy of convex (without the gradient) 5 N. Cesa-Bianchi (UNIMI) Online Approach to ML 2 / 53

  3. Summary My beautiful regret 1 A supposedly fun game I’ll play again 2 A graphic novel 3 The joy of convex 4 The joy of convex (without the gradient) 5 N. Cesa-Bianchi (UNIMI) Online Approach to ML 3 / 53

  4. Machine learning Classification / regression tasks Predictive models h mapping data instances X to labels Y (e.g., binary classifier) � � Training data S T = ( X 1 , Y 1 ) , . . . , ( X T , Y T ) (e.g., email messages with spam vs. nonspam annotations) Learning algorithm A (e.g., Support Vector Machine) maps training data S T to model h = A ( S T ) Evaluate the risk of the trained model h with respect to a given loss function N. Cesa-Bianchi (UNIMI) Online Approach to ML 4 / 53

  5. Two notions of risk View data as a statistical sample: statistical risk � �� � loss A ( S T ) , ( X , Y ) E � ���� �� ���� � � �� �� �� � test trained example model � � Training set S T = ( X 1 , Y 1 ) , . . . , ( X T , Y T ) and test example ( X , Y ) drawn i.i.d. from the same unknown and fixed distribution View data as an arbitrary sequence: sequential risk T � � � loss A ( S t − 1 ) , ( X t , Y t ) � ������ �� ������ � � ����� �� ����� � t = 1 test trained example model Sequence of models trained on growing prefixes � � S t = ( X 1 , Y 1 ) , . . . , ( X t , Y t ) of the data sequence N. Cesa-Bianchi (UNIMI) Online Approach to ML 5 / 53

  6. Regrets, I had a few Learning algorithm A maps datasets to models in a given class H Variance error in statistical learning � �� � �� � � E loss A ( S T ) , ( X , Y ) − inf h ∈ H E loss h , ( X , Y ) compare to expected loss of best model in the class Regret in online learning T T � � � � � � loss A ( S t − 1 ) , ( X t , Y t ) − inf loss h , ( X t , Y t ) h ∈ H t = 1 t = 1 compare to cumulative loss of best model in the class N. Cesa-Bianchi (UNIMI) Online Approach to ML 6 / 53

  7. Incremental model update A natural blueprint for online learning algorithms For t = 1, 2, . . . Apply current model h t − 1 to next data element ( X t , Y t ) 1 Update current model: h t − 1 → h t ∈ H 2 Goal: control regret T T � � � � � � loss h t − 1 , ( X t , Y t ) − inf loss h , ( X t , Y t ) h ∈ H t = 1 t = 1 View this as a repeated game between a player generating predictors h t ∈ H and an opponent generating data ( X t , Y t ) N. Cesa-Bianchi (UNIMI) Online Approach to ML 7 / 53

  8. Summary My beautiful regret 1 A supposedly fun game I’ll play again 2 A graphic novel 3 The joy of convex 4 The joy of convex (without the gradient) 5 N. Cesa-Bianchi (UNIMI) Online Approach to ML 8 / 53

  9. Theory of repeated games James Hannan David Blackwell (1922–2010) (1919–2010) Learning to play a game (1956) Play a game repeatedly against a possibly suboptimal opponent N. Cesa-Bianchi (UNIMI) Online Approach to ML 9 / 53

  10. Zero-sum 2-person games played more than once 1 2 . . . M N × M known loss matrix ℓ ( 1, 1 ) ℓ ( 1, 2 ) 1 . . . Row player (player) 2 ℓ ( 2, 1 ) ℓ ( 2, 2 ) . . . has N actions . . . ... . . . . . . Column player (opponent) N has M actions For each game round t = 1, 2, . . . Player chooses action i t and opponent chooses action y t The player su ff ers loss ℓ ( i t , y t ) ( = gain of opponent) Player can learn from opponent’s history of past choices y 1 , . . . , y t − 1 N. Cesa-Bianchi (UNIMI) Online Approach to ML 10 / 53

  11. Prediction with expert advice t = 1 t = 2 . . . 1 ℓ 1 ( 1 ) ℓ 2 ( 1 ) . . . ℓ 1 ( 2 ) ℓ 2 ( 2 ) 2 . . . . . . ... . . . . . . ℓ 1 ( N ) ℓ 2 ( N ) N Volodya Vovk Manfred Warmuth Opponent’s moves y 1 , y 2 , . . . define a sequential prediction problem with a time-varying loss function ℓ ( i t , y t ) = ℓ t ( i t ) N. Cesa-Bianchi (UNIMI) Online Approach to ML 11 / 53

  12. Playing the experts game N actions ? ? ? ? ? ? ? ? ? For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) N. Cesa-Bianchi (UNIMI) Online Approach to ML 12 / 53

  13. Playing the experts game N actions ? ? ? ? ? ? ? ? ? For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) Player picks an action I t (possibly using randomization) and 2 incurs loss ℓ t ( I t ) N. Cesa-Bianchi (UNIMI) Online Approach to ML 12 / 53

  14. Playing the experts game N actions 7 3 2 4 1 6 7 4 9 For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) Player picks an action I t (possibly using randomization) and 2 incurs loss ℓ t ( I t ) Player gets feedback information: ℓ t ( 1 ) , . . . , ℓ t ( N ) 3 N. Cesa-Bianchi (UNIMI) Online Approach to ML 12 / 53

  15. Oblivious opponents Losses ℓ t ( 1 ) , . . . , ℓ t ( N ) for all t = 1, 2, . . . are fixed beforehand, and unknown to the (randomized) player Oblivious regret minimization � T � T � � def ℓ t ( i ) want R T = E ℓ t ( I t ) − min = o ( T ) i = 1,..., N t = 1 t = 1 N. Cesa-Bianchi (UNIMI) Online Approach to ML 13 / 53

  16. Bounds on regret [Experts’ paper, 1997] Lower bound using random losses ℓ t ( i ) → L t ( i ) ∈ { 0, 1 } independent random coin flip � T � = T � For any player strategy E L t ( I t ) 2 t = 1 Then the expected regret is � �� � � � 1 T � T ln N � E max 2 − L t ( i ) = 1 − o ( 1 ) 2 i = 1,..., N t = 1 N. Cesa-Bianchi (UNIMI) Online Approach to ML 14 / 53

  17. Exponentially weighted forecaster At time t pick action I t = i with probability proportional to � � t − 1 � exp − η ℓ s ( i ) s = 1 the sum at the exponent is the total loss of action i up to now Regret bound [Experts’ paper, 1997] � � T ln N If η = ( ln N ) / ( 8 T ) then R T � 2 Matching lower bound including constants � Dynamic choice η t = ( ln N ) / ( 8 t ) only loses small constants N. Cesa-Bianchi (UNIMI) Online Approach to ML 15 / 53

  18. The bandit problem: playing an unknown game N actions ? ? ? ? ? ? ? ? ? For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) N. Cesa-Bianchi (UNIMI) Online Approach to ML 16 / 53

  19. The bandit problem: playing an unknown game N actions ? ? ? ? ? ? ? ? ? For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) Player picks an action I t (possibly using randomization) and 2 incurs loss ℓ t ( I t ) N. Cesa-Bianchi (UNIMI) Online Approach to ML 16 / 53

  20. The bandit problem: playing an unknown game N actions ? 3 ? ? ? ? ? ? ? For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) Player picks an action I t (possibly using randomization) and 2 incurs loss ℓ t ( I t ) Player gets feedback information: Only ℓ t ( I t ) is revealed 3 N. Cesa-Bianchi (UNIMI) Online Approach to ML 16 / 53

  21. The bandit problem: playing an unknown game N actions ? 3 ? ? ? ? ? ? ? For t = 1, 2, . . . Loss ℓ t ( i ) ∈ [ 0, 1 ] is assigned to every action i = 1, . . . , N 1 (hidden from the player) Player picks an action I t (possibly using randomization) and 2 incurs loss ℓ t ( I t ) Player gets feedback information: Only ℓ t ( I t ) is revealed 3 Many applications Ad placement, dynamic content adaptation, routing, online auctions N. Cesa-Bianchi (UNIMI) Online Approach to ML 16 / 53

  22. Summary My beautiful regret 1 A supposedly fun game I’ll play again 2 A graphic novel 3 The joy of convex 4 The joy of convex (without the gradient) 5 N. Cesa-Bianchi (UNIMI) Online Approach to ML 17 / 53

  23. Relationships between actions [Mannor and Shamir, 2011] Undirected Directed N. Cesa-Bianchi (UNIMI) Online Approach to ML 18 / 53

  24. A graph of relationships over actions ? ? ? ? ? ? ? ? ? ? N. Cesa-Bianchi (UNIMI) Online Approach to ML 19 / 53

  25. A graph of relationships over actions ? ? ? ? ? ? ? ? ? ? N. Cesa-Bianchi (UNIMI) Online Approach to ML 19 / 53

  26. A graph of relationships over actions 7 3 6 7 2 ? ? ? ? ? N. Cesa-Bianchi (UNIMI) Online Approach to ML 19 / 53

  27. Recovering expert and bandit settings Experts: clique Bandits: empty graph 7 ? 3 6 3 ? 7 2 1 2 ? ? ? ? 4 9 ? ? 4 ? N. Cesa-Bianchi (UNIMI) Online Approach to ML 20 / 53

Recommend

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING =

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING =

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING = call = ? = crawl Training Test Data Data Train call Apply Model Label 2 STANDARD MACHINE LEARNING SETTING Predicting the future

746 views • 51 slides
Online machine learning with decision trees Max Halford University of Toulouse Online machine

Online machine learning with decision trees Max Halford University of Toulouse Online machine

Online machine learning with decision trees Max Halford University of Toulouse Online machine learning with decision trees Max Halford 1 / 46 Thursday 7 th May, 2020 Decision trees Most successful general-purpose algorithm in modern

698 views • 46 slides
Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization

Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization

CS6501: T opics in Learning and Game Theory (Fall 2019) Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization Measure Algorithm Performance via Regret Warm-up: A Simple Example 2 Overview of Machine

629 views • 37 slides
A machine learning approach against a masked AES L. LERMAN , S. FERNANDES MEDEIROS, G. BONTEMPI,

A machine learning approach against a masked AES L. LERMAN , S. FERNANDES MEDEIROS, G. BONTEMPI,

A machine learning approach against a masked AES A machine learning approach against a masked AES L. LERMAN , S. FERNANDES MEDEIROS, G. BONTEMPI, and O. MARKOWITCH Universit Libre de Bruxelles Faculty of Sciences Department of Computer

349 views • 20 slides
Machine Learning - MT 2017 7 Bayesian Approach to Machine Learning Christoph Haase University of

Machine Learning - MT 2017 7 Bayesian Approach to Machine Learning Christoph Haase University of

Machine Learning - MT 2017 7 Bayesian Approach to Machine Learning Christoph Haase University of Oxford October 23, 2017 Frequentist vs Bayesian Approaches Different views on probability: Frequentists: Probability of an event represents

469 views • 32 slides
Bayesian Learning l A powerful approach in machine learning l Combine data seen so far with prior

Bayesian Learning l A powerful approach in machine learning l Combine data seen so far with prior

Bayesian Learning l A powerful approach in machine learning l Combine data seen so far with prior beliefs This is what has allowed us to do machine learning, have good inductive biases, overcome "No free lunch", and obtain good

1.42k views • 29 slides
Learning to de-anonymize social networks A machine learning approach to social graph

Learning to de-anonymize social networks A machine learning approach to social graph

Learning to de-anonymize social networks A machine learning approach to social graph de-anonymization Kumar Sharad October 28, 2016 Royal Holloway, University of London ACM Workshop on Artificial Intelligence and Security, Vienna, Austria

1.01k views • 69 slides
The Probabilistic Approach to Learning from Data Prob. Readings: Matt Gormley

The Probabilistic Approach to Learning from Data Prob. Readings: Matt Gormley

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University The Probabilistic Approach to Learning from Data Prob. Readings: Matt

736 views • 58 slides
Online Learning Your guide: Avrim Blum Carnegie Mellon University [Machine Learning Summer School

Online Learning Your guide: Avrim Blum Carnegie Mellon University [Machine Learning Summer School

Online Learning Your guide: Avrim Blum Carnegie Mellon University [Machine Learning Summer School 2012] Itinerary Stop 1: Minimizing regret and combining advice. Randomized Wtd Majority / Multiplicative Weights alg Connections to game

538 views • 37 slides
Online Learning & Margins Instructor: Sham Kakade 1 Introduction There are two common

Online Learning & Margins Instructor: Sham Kakade 1 Introduction There are two common

CSE 546: Machine Learning Lecture 9 Online Learning & Margins Instructor: Sham Kakade 1 Introduction There are two common models of study: Online Learning No assumptions about data generating process. Worst case analysis. Fundamental

61 views • 3 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine Learning (las.ethz.ch) What is Machine Learning I: An example Classify email messages as Spam or Non Spam Classical Approach : manual rules

1.27k views • 59 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

559 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Online Joint GlueX-EIC-PANDA Machine Learning Workshop Machine Learning for Beginners Thomas

Online Joint GlueX-EIC-PANDA Machine Learning Workshop Machine Learning for Beginners Thomas

Online Joint GlueX-EIC-PANDA Machine Learning Workshop Machine Learning for Beginners Thomas Stibor GSI Helmholtzzentrum f ur Schwerionenforschung GmbH t.stibor@gsi.de 21 th September 2020 - 25 th September 2020 21 th September 2020 - 25 th

937 views • 26 slides
A Machine Learning Approach A Machine Learning Approach A Machine Learning Approach A Machine

A Machine Learning Approach A Machine Learning Approach A Machine Learning Approach A Machine

A Machine Learning Approach A Machine Learning Approach A Machine Learning Approach A Machine Learning Approach to Preventing to Preventing to Preventing to Preventing Avoidable ED Utilization Avoidable ED Utilization Avoidable ED

727 views • 13 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

401 views • 26 slides
Machine Learning for Sequence Learning Learning in an All-Subsequence Space Severin Gsponer,

Machine Learning for Sequence Learning Learning in an All-Subsequence Space Severin Gsponer,

Machine Learning for Sequence Learning Learning in an All-Subsequence Space Severin Gsponer, Georgiana Ifrim, Barry Smyth January 20, 2016 Outline Background Linear Classifiers for Sequences SEQL Approach Contribution Future

298 views • 18 slides
5.1 Online versus Offline SVMs We start with a review of the Offline Support Vector Machine.

5.1 Online versus Offline SVMs We start with a review of the Offline Support Vector Machine.

CS/CNS/EE 253: Advanced Topics in Machine Learning Topic: Running time analysis for Offline and Online Optimization Lecturer: Andreas Krause Scribe: Chris Kennelly Date: Jan. 20, 2010 5.1 Online versus Offline SVMs We start with a review of the

443 views • 4 slides
Teaching with Online Platforms What is an Online Learning Platform? A n Online Learning Platform

Teaching with Online Platforms What is an Online Learning Platform? A n Online Learning Platform

Teaching with Online Platforms What is an Online Learning Platform? A n Online Learning Platform is typically a website used by educational institutions as a tool or primary environment for learning. It can serve many functions, such as being a

912 views • 9 slides
Machine learning theory On line learning Hamid Beigy Sharif university of technology May 16,

Machine learning theory On line learning Hamid Beigy Sharif university of technology May 16,

Machine learning theory On line learning Hamid Beigy Sharif university of technology May 16, 2020 Table of contents 1. Introduction 2. Online classification in the realizable case 3. Online classification in the unrealizable case 4.

1.02k views • 57 slides
Discussion of Benefits for student learning 1. Increase in and an equalisation of engagement Quiz

Discussion of Benefits for student learning 1. Increase in and an equalisation of engagement Quiz

How can student interactivity be enhanced through the use of a blended learning approach? A Blended Learning Model using Outline 1) Online Survey [Questionnaire] 2) About the Project 3) Experiencing Moodle [Participants online] 4) Moodle

287 views • 5 slides
Online Learning Lorenzo Rosasco MIT, 9.520 L. Rosasco Online Learning About this class Goal

Online Learning Lorenzo Rosasco MIT, 9.520 L. Rosasco Online Learning About this class Goal

Online Learning Lorenzo Rosasco MIT, 9.520 L. Rosasco Online Learning About this class Goal To introduce theory and algorithms for online learning. L. Rosasco Online Learning Plan Different views on online learning From batch to online

620 views • 43 slides
Prediction without probability: a PDE approach to a model problem from machine learning Robert V.

Prediction without probability: a PDE approach to a model problem from machine learning Robert V.

Prediction without probability: a PDE approach to a model problem from machine learning Robert V. Kohn Courant Institute, NYU Joint work with Kangping Zhu (PhD 2014) and Nadejda Drenska (in progress) Mathematics for Nonlinear Phenomena:

924 views • 44 slides

More recommend