suboptimality of penalized empirical risk minimization in
play

Suboptimality of Penalized Empirical Risk Minimization in - PowerPoint PPT Presentation

Jul 30, 2023 •161 likes •708 views

Suboptimality of Penalized Empirical Risk Minimization in Classification. Guillaume Lecu e Universit e Paris 6 COLT 2007, June 13 General Framework. Aggregations Procedures. Optimality in classification. Motivation. M prior estimators

  1. Suboptimality of Penalized Empirical Risk Minimization in Classification. Guillaume Lecu´ e Universit´ e Paris 6 COLT 2007, June 13

  2. General Framework. Aggregations Procedures. Optimality in classification. Motivation. M prior estimators (’weak’ estimators) : f 1 , . . . , f M n observations : D n Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  3. General Framework. Aggregations Procedures. Optimality in classification. Motivation. M prior estimators (’weak’ estimators) : f 1 , . . . , f M n observations : D n Aim Construction of a new estimator which is approximatively as good as the best ’weak’ estimator : Aggregation method or Aggregate Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  4. General Framework. Aggregations Procedures. Optimality in classification. Examples. Adaptation : Observations : D m + n Estimation : D m → non-adaptive estimators f 1 , . . . , f M . learning : D ( n ) → aggregate ˜ f n (adaptive). Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  5. General Framework. Aggregations Procedures. Optimality in classification. Examples. Adaptation : Observations : D m + n Estimation : D m → non-adaptive estimators f 1 , . . . , f M . learning : D ( n ) → aggregate ˜ f n (adaptive). Estimation : ǫ − net : f 1 , . . . , f M (functions) learning : D n → aggregate ˜ f n . Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  6. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  7. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  8. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {− 1 , 1 } ? Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  9. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {− 1 , 1 } ? f : X �− → {− 1 , 1 } : prediction rule. Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  10. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {− 1 , 1 } ? f : X �− → {− 1 , 1 } : prediction rule. Bayes risk : A 0 ( f ) = P [ f ( X ) � = Y ] Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  11. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {− 1 , 1 } ? f : X �− → {− 1 , 1 } : prediction rule. Bayes risk : A 0 ( f ) = P [ f ( X ) � = Y ] Bayes rule : f ∗ ( x ) = Sign (2 η ( x ) − 1) where η ( x ) = P [ Y = 1 | X = x ] . def A ∗ = min f A 0 ( f ) = A 0 ( f ∗ ) 0 Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  12. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {− 1 , 1 } ? f : X �− → {− 1 , 1 } : prediction rule. Bayes risk : A 0 ( f ) = P [ f ( X ) � = Y ] Bayes rule : f ∗ ( x ) = Sign (2 η ( x ) − 1) where η ( x ) = P [ Y = 1 | X = x ] . def A ∗ = min f A 0 ( f ) = A 0 ( f ∗ ) 0 Prediction → estimation : estimation of f ∗ . Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  13. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( X , A ) a measurable space, ( X , Y ) ∼ π valued in X × {− 1 , 1 } , D n = (( X 1 , Y 1 ) , . . . , ( X n , Y n )) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {− 1 , 1 } ? f : X �− → {− 1 , 1 } : prediction rule. Bayes risk : A 0 ( f ) = P [ f ( X ) � = Y ] Bayes rule : f ∗ ( x ) = Sign (2 η ( x ) − 1) where η ( x ) = P [ Y = 1 | X = x ] . def A ∗ = min f A 0 ( f ) = A 0 ( f ∗ ) 0 Prediction → estimation : estimation of f ∗ . excess risk : A 0 ( f ) − A ∗ 0 Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  14. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( f : X �− → R ) → risk A 0 ( f ) = E [ φ 0 ( Yf ( X ))] where φ 0 ( x ) = 1 I ( x ≤ 0) classical loss or 0 − 1 loss Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  15. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( f : X �− → R ) → risk A 0 ( f ) = E [ φ 0 ( Yf ( X ))] where φ 0 ( x ) = 1 I ( x ≤ 0) classical loss or 0 − 1 loss φ 1 ( x ) = max(0 , 1 − x ) hinge loss or (SVM loss) x �− → log 2 (1 + exp( − x )) ’Logit-Boosting’ loss x �− → exp( − x ) exponential Boosting loss → (1 − x ) 2 x �− quadratic loss → max(0 , 1 − x ) 2 x �− 2-norm soft margin loss Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  16. General Framework. Aggregations Procedures. Optimality in classification. Model of classification ( f : X �− → R ) → risk A 0 ( f ) = E [ φ 0 ( Yf ( X ))] where φ 0 ( x ) = 1 I ( x ≤ 0) classical loss or 0 − 1 loss φ 1 ( x ) = max(0 , 1 − x ) hinge loss or (SVM loss) x �− → log 2 (1 + exp( − x )) ’Logit-Boosting’ loss x �− → exp( − x ) exponential Boosting loss → (1 − x ) 2 x �− quadratic loss → max(0 , 1 − x ) 2 x �− 2-norm soft margin loss A φ ∗ def φ − risk : A φ ( f ) = E [ φ ( Yf ( X ))] , = inf f A ( f ) = A ( f φ ∗ ) , excess φ − risk : A φ ( f ) − A φ ∗ . n n ( f ) = 1 � empirical φ − risk : A φ φ ( Y i f ( X i )) . n i =1 Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  17. General Framework. Aggregations Procedures. Optimality in classification. Selectors. φ : R �− → R a loss, F 0 = { f 1 , . . . , f M } ⊂ F a dictionary. Empirical Risk Minimization (ERM) :(Vapnik, Chervonenkis...) ˜ f ERM f ∈F 0 A φ ∈ Arg min n ( f ) . n Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  18. General Framework. Aggregations Procedures. Optimality in classification. Selectors. φ : R �− → R a loss, F 0 = { f 1 , . . . , f M } ⊂ F a dictionary. Empirical Risk Minimization (ERM) :(Vapnik, Chervonenkis...) ˜ f ERM f ∈F 0 A φ ∈ Arg min n ( f ) . n penalized Empirical Risk Minimization (pERM) : ˜ f ∈F 0 [ A φ f ERM ∈ Arg min n ( f ) + pen ( f )] , n where pen is a penalty function. (Barron, Bartlett, Birg´ e, Boucheron, Koltchinski, Lugosi, Massart,...) Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  19. General Framework. Aggregations Procedures. Optimality in classification. Aggregation methods with exponential weights. φ : R �− → R a loss, F 0 = { f 1 , . . . , f M } ⊂ F a dictionary. Aggregate with Exponential weights (AEW) : � − nTA φ � exp n ( f ) ˜ � w ( n ) T ( f ) f , where w ( n ) f AEW = T ( f ) = � , n , T � − nTA φ � g ∈F 0 exp n ( g ) f ∈F 0 T − 1 : temperature parameter. Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

  20. General Framework. Aggregations Procedures. Optimality in classification. Aggregation methods with exponential weights. φ : R �− → R a loss, F 0 = { f 1 , . . . , f M } ⊂ F a dictionary. Aggregate with Exponential weights (AEW) : � − nTA φ � exp n ( f ) ˜ � w ( n ) T ( f ) f , where w ( n ) f AEW = T ( f ) = � , n , T � − nTA φ � g ∈F 0 exp n ( g ) f ∈F 0 T − 1 : temperature parameter. Cumulative Aggregate with Exponential Weights (CAEW) :(Catoni, Yang,...) n = 1 ˜ � ˜ f CAEW f AEW . n , T k , T n k =1 Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

Recommend

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view Perceptron CRF Notation for Linear Models Training data: {(x 1 , y 1 ), (x 2 , y 2 ), , (x N , y N )} Testing data: {(x N+1 , y N+1 ),

418 views • 21 slides
CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical

CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical

CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical Risk Minimization Learning: Reminder Known data universe X and an unknown probability distribution D on X Known concept class C and an unknown

341 views • 17 slides
High Order Methods for Empirical Risk Minimization Alejandro Ribeiro Department of Electrical and

High Order Methods for Empirical Risk Minimization Alejandro Ribeiro Department of Electrical and

High Order Methods for Empirical Risk Minimization Alejandro Ribeiro Department of Electrical and Systems Engineering University of Pennsylvania aribeiro@seas.upenn.edu Thanks to: Aryan Mokhtari, Mark Eisen, ONR, NSF DIMACS Workshop on

775 views • 40 slides
Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang ,

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang ,

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang , Changyou Chen and Jinhui Xu State University of New York at Bu ff alo International Conference on Machine Learning 2019 Di Wang Non-convex DP-ERM ICML

544 views • 27 slides
WEIGHTED SUMS OF RANDOM KITCHEN SINKS Replacing minimization with randomization in learning

WEIGHTED SUMS OF RANDOM KITCHEN SINKS Replacing minimization with randomization in learning

WEIGHTED SUMS OF RANDOM KITCHEN SINKS Replacing minimization with randomization in learning The model Given a set of training data in a domain Fit a function to minimize risk Empirical Risk Risk Loss Function Hinge loss

621 views • 13 slides
Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik

Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik

Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik Empirical Risk Minimization n x R d f ( x ) := 1 X min f i ( x ) n i =1 Empirical Risk Minimization n x R d f ( x ) := 1 X min f i (

1.07k views • 28 slides
Introduction to Machine Learning CMU-10701 10. Risk Minimization Barnabs Pczos 10. Risk

Introduction to Machine Learning CMU-10701 10. Risk Minimization Barnabs Pczos 10. Risk

Introduction to Machine Learning CMU-10701 10. Risk Minimization Barnabs Pczos 10. Risk Minimization 2 What have we seen so far? Several algorithms that seem to work fine on training datasets: Linear regression Nave Bayes

1.13k views • 51 slides
Introduction to Machine Learning Vapnik Chervonenkis Theory Barnabs Pczos Empirical Risk

Introduction to Machine Learning Vapnik Chervonenkis Theory Barnabs Pczos Empirical Risk

Introduction to Machine Learning Vapnik Chervonenkis Theory Barnabs Pczos Empirical Risk and True Risk 2 Empirical Risk Shorthand: True risk of f (deterministic) : Bayes risk : Let us use the empirical counter part: Empirical risk: 3

1.46k views • 69 slides
H OW C AN W E D ESIGN AN A LGORITHM ? A possible way to do this is penalized empirical risk

H OW C AN W E D ESIGN AN A LGORITHM ? A possible way to do this is penalized empirical risk

R EGULARIZATION FOR M ULTI -O UTPUT L EARNING Francesca Odone and Lorenzo Rosasco RegML 2013 Regularization Methods for High Dimensional Learning Regularization for Multi-Output Learning A BOUT THIS CLASS G OAL In many practical problems, it is

669 views • 44 slides
Empirical Loss Minimization Traffic sign - STOP Sample i.i.d. points Stochastic

Empirical Loss Minimization Traffic sign - STOP Sample i.i.d. points Stochastic

Empirical Loss Minimization Traffic sign - STOP Sample i.i.d. points Stochastic Gradient Descent Lon Bottou, Frank E Curtis, Jorge Nocedal Optimization methods for large-scale machine learning SVRG:

738 views • 39 slides
Introduction to Machine Learning ML-Basics: Losses & Risk Minimization Learning goals Know

Introduction to Machine Learning ML-Basics: Losses & Risk Minimization Learning goals Know

Introduction to Machine Learning ML-Basics: Losses & Risk Minimization Learning goals Know the concept of loss Understand the relationship between loss and risk Understand the relationship between risk minimization and finding the best

645 views • 10 slides
On the Local Minima of the Empirical Risk Chi Jin * 1 , Lydia T. Liu* 1 , Rong Ge 2 , Michael I.

On the Local Minima of the Empirical Risk Chi Jin * 1 , Lydia T. Liu* 1 , Rong Ge 2 , Michael I.

On the Local Minima of the Empirical Risk Chi Jin * 1 , Lydia T. Liu* 1 , Rong Ge 2 , Michael I. Jordan 1 1EECS, University of California, Berkeley. 2Duke University. 1 / 6 Chi Jin On the Local Minima of the Empirical Risk Overview Nonconvex

425 views • 14 slides
Introduction to Machine Learning Risk Minimization Barnabs Pczos What have we seen so far?

Introduction to Machine Learning Risk Minimization Barnabs Pczos What have we seen so far?

Introduction to Machine Learning Risk Minimization Barnabs Pczos What have we seen so far? Several classification & regression algorithms seem to work fine on training datasets: Linear regression Gaussian Processes Nave

802 views • 53 slides
Minimization Using Descent Information we will consider the minimization of unconstrained

Minimization Using Descent Information we will consider the minimization of unconstrained

Minimization Using Descent Information we will consider the minimization of unconstrained functions of several variables where we now assume we have some derivative information such as the gradient vector or the Hessian matrix. Recall

131 views • 11 slides
The Many Flavors of Penalized Linear Discriminant Analysis Daniela M. Witten Assistant Professor

The Many Flavors of Penalized Linear Discriminant Analysis Daniela M. Witten Assistant Professor

Linear Discriminant Analysis Penalized LDA Connections The Many Flavors of Penalized Linear Discriminant Analysis Daniela M. Witten Assistant Professor of Biostatistics University of Washington May 9, 2011 Fourth Erich L. Lehmann Symposium

698 views • 40 slides
An Empirical Decomposition of Risk and Liquidity in Nominal and Inf lation- Indexed

An Empirical Decomposition of Risk and Liquidity in Nominal and Inf lation- Indexed

An Empirical Decomposition of Risk and Liquidity in Nominal and Inf lation- Indexed Government Bonds Carolin Pflueger, Harvard Business School Luis Viceira, Harvard Business School November 8, 2011 Bond Risk Unlike single stocks,

949 views • 47 slides
ON THE OPTIMIZATION LANDSCAPE OF NEURAL NETWORKS JOAN BRUNA , CIMS + CDS, NYU in collaboration

ON THE OPTIMIZATION LANDSCAPE OF NEURAL NETWORKS JOAN BRUNA , CIMS + CDS, NYU in collaboration

ON THE OPTIMIZATION LANDSCAPE OF NEURAL NETWORKS JOAN BRUNA , CIMS + CDS, NYU in collaboration with D.Freeman (UC Berkeley), Luca Venturi & Afonso Bandeira (NYU) MOTIVATION We consider the standard Empirical Risk Minimization setup: `

683 views • 46 slides
Manifold Identification for Ultimately Communication-Efficient Distributed Optimization Yu-Sheng

Manifold Identification for Ultimately Communication-Efficient Distributed Optimization Yu-Sheng

Manifold Identification for Ultimately Communication-Efficient Distributed Optimization Yu-Sheng Li Joint work with Wei-Lin Chiang (NTU) and Ching-pei Lee (NUS) Outline Overview Empirical Risk Minimization The Proposed Algorithm Experiments

858 views • 49 slides
The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016

The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016

The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016 Joint work with Yu Bai and Andrea Montanari Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 1 / 17 Binary linear

341 views • 33 slides
Bayesian Counterfactual Risk Minimization Ben London (blondon@) Amazon Music Ted Sandler (sandler@)

Bayesian Counterfactual Risk Minimization Ben London (blondon@) Amazon Music Ted Sandler (sandler@)

Bayesian Counterfactual Risk Minimization Ben London (blondon@) Amazon Music Ted Sandler (sandler@) Amazon Music International Conference on Machine Learning Long Beach, CA, June 11, 2019 Learning from Logged Data Pull log data e.g., user i

593 views • 10 slides
CS 6316 Machine Learning Introduction to Learning Theory Yangfeng Ji Department of Computer

CS 6316 Machine Learning Introduction to Learning Theory Yangfeng Ji Department of Computer

CS 6316 Machine Learning Introduction to Learning Theory Yangfeng Ji Department of Computer Science University of Virginia Overview 1. A Toy Example 2. A Formal Model 3. Empirical Risk Minimization 4. Finite Hypothesis Classes 5. PAC

1.08k views • 72 slides
Machine Learning Theory CS 446 1. SVM risk SVM risk Consider the empirical and true/population

Machine Learning Theory CS 446 1. SVM risk SVM risk Consider the empirical and true/population

Machine Learning Theory CS 446 1. SVM risk SVM risk Consider the empirical and true/population risk of SVM: given f , R ( f ) = 1 ( Y ( y i R ( f ) = E f ( X )) , f ( x i )) , n i =1 and furthermore de fi ne

1.1k views • 77 slides
Minimization Satoru Iwata (University of Tokyo) Submodular Function Minimization ( )

Minimization Satoru Iwata (University of Tokyo) Submodular Function Minimization ( )

Submodular Function Minimization Satoru Iwata (University of Tokyo) Submodular Function Minimization ( ) 0 f Assumption: X Minimization Evaluation Algorithm Oracle ( X ) f Minimizer min{ ( ) | } ? f Y Y V

177 views • 16 slides
Suboptimality of Asian Executive Indexed Options Carole Bernard Phelim Boyle Jit Seng Chen

Suboptimality of Asian Executive Indexed Options Carole Bernard Phelim Boyle Jit Seng Chen

Suboptimality of Asian Executive Indexed Options Carole Bernard Phelim Boyle Jit Seng Chen Actuarial Research Conference August 13, 2011 Outline 1. Options Preliminaries 2. Assumptions 3. Asian Executive Indexed Option 4. Cost-Efficiency 5.

467 views • 18 slides

More recommend