the landscape of empirical risk for non convex losses
play

The landscape of empirical risk for non-convex losses Song Mei - PowerPoint PPT Presentation

Apr 10, 2024 •11 likes •341 views

The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016 Joint work with Yu Bai and Andrea Montanari Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 1 / 17 Binary linear

  1. The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016 Joint work with Yu Bai and Andrea Montanari Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 1 / 17

  2. Binary linear classification The model Z ✐ ❂ ✭ X ✐ ❀ ❨ ✐ ✮ . X ✐ ✷ R ❞ , ❨ ✐ ✷ ❢ ✵ ❀ ✶ ❣ , i = 1,. . . , n. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 2 / 17

  3. Non-convex formulation of binary classification The model Z ✐ ❂ ✭ X ✐ ❀ ❨ ✐ ✮ . X ✐ ✷ R ❞ , ❨ ✐ ✷ ❢ ✵ ❀ ✶ ❣ , i = 1,. . . , n. ◮ Convex logit loss ( ❵ ❝ is cvx in θ ) ✏ ✑ ❵ ❝ ✭ θ ❀ Z ✮ ❂ ❨ ❤ X ❀ θ ✐ � ❧♦❣ ✶ ✰ ❡①♣✭ ❤ X ❀ θ ✐ ✮ ✿ ◮ Non-convex loss ( ❵ is not cvx in θ ) ✏ ✑ ✷ ❵ ✭ θ ❀ Z ✮ ❂ ❨ � ✛ ✭ ❤ X ❀ θ ✐ ✮ ❀ where ✛ ✭ t ✮ ❂ ✶ ❂ ✭✶ ✰ ❡①♣✭ t ✮✮ ✿ ◮ Empirical Risk ♥ ❳ ❘ ♥ ✭ θ ✮ ❂ ✶ ❜ ❵ ✭ θ ❀ Z ✐ ✮ ✿ ♥ ✐ ❂✶ ◮ Empirical risk minimizer ❫ ❜ θ ♥ ❂ ❛r❣ ♠✐♥ ❘ ♥ ✭ θ ✮ ✿ θ ✷ B ❞ ✭ ❘ ✮ Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 3 / 17

  4. Non-convex formulation of binary classification The model Z ✐ ❂ ✭ X ✐ ❀ ❨ ✐ ✮ . X ✐ ✷ R ❞ , ❨ ✐ ✷ ❢ ✵ ❀ ✶ ❣ , i = 1,. . . , n. ◮ Convex logit loss ( ❵ ❝ is cvx in θ ) ✏ ✑ ❵ ❝ ✭ θ ❀ Z ✮ ❂ ❨ ❤ X ❀ θ ✐ � ❧♦❣ ✶ ✰ ❡①♣✭ ❤ X ❀ θ ✐ ✮ ✿ ◮ Non-convex loss ( ❵ is not cvx in θ ) ✏ ✑ ✷ ❵ ✭ θ ❀ Z ✮ ❂ ❨ � ✛ ✭ ❤ X ❀ θ ✐ ✮ ❀ where ✛ ✭ t ✮ ❂ ✶ ❂ ✭✶ ✰ ❡①♣✭ t ✮✮ ✿ ◮ Empirical Risk ♥ ❳ ❘ ♥ ✭ θ ✮ ❂ ✶ ❜ ❵ ✭ θ ❀ Z ✐ ✮ ✿ ♥ ✐ ❂✶ ◮ Empirical risk minimizer ❫ ❜ θ ♥ ❂ ❛r❣ ♠✐♥ ❘ ♥ ✭ θ ✮ ✿ θ ✷ B ❞ ✭ ❘ ✮ Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 3 / 17

  5. Non-convex formulation of binary classification The model Z ✐ ❂ ✭ X ✐ ❀ ❨ ✐ ✮ . X ✐ ✷ R ❞ , ❨ ✐ ✷ ❢ ✵ ❀ ✶ ❣ , i = 1,. . . , n. ◮ Convex logit loss ( ❵ ❝ is cvx in θ ) ✏ ✑ ❵ ❝ ✭ θ ❀ Z ✮ ❂ ❨ ❤ X ❀ θ ✐ � ❧♦❣ ✶ ✰ ❡①♣✭ ❤ X ❀ θ ✐ ✮ ✿ ◮ Non-convex loss ( ❵ is not cvx in θ ) ✏ ✑ ✷ ❵ ✭ θ ❀ Z ✮ ❂ ❨ � ✛ ✭ ❤ X ❀ θ ✐ ✮ ❀ where ✛ ✭ t ✮ ❂ ✶ ❂ ✭✶ ✰ ❡①♣✭ t ✮✮ ✿ ◮ Empirical Risk ♥ ❳ ❘ ♥ ✭ θ ✮ ❂ ✶ ❜ ❵ ✭ θ ❀ Z ✐ ✮ ✿ ♥ ✐ ❂✶ ◮ Empirical risk minimizer ❫ ❜ θ ♥ ❂ ❛r❣ ♠✐♥ ❘ ♥ ✭ θ ✮ ✿ θ ✷ B ❞ ✭ ❘ ✮ Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 3 / 17

  6. Non-convex formulation of binary classification The model Z ✐ ❂ ✭ X ✐ ❀ ❨ ✐ ✮ . X ✐ ✷ R ❞ , ❨ ✐ ✷ ❢ ✵ ❀ ✶ ❣ , i = 1,. . . , n. ◮ Convex logit loss ( ❵ ❝ is cvx in θ ) ✏ ✑ ❵ ❝ ✭ θ ❀ Z ✮ ❂ ❨ ❤ X ❀ θ ✐ � ❧♦❣ ✶ ✰ ❡①♣✭ ❤ X ❀ θ ✐ ✮ ✿ ◮ Non-convex loss ( ❵ is not cvx in θ ) ✏ ✑ ✷ ❵ ✭ θ ❀ Z ✮ ❂ ❨ � ✛ ✭ ❤ X ❀ θ ✐ ✮ ❀ where ✛ ✭ t ✮ ❂ ✶ ❂ ✭✶ ✰ ❡①♣✭ t ✮✮ ✿ ◮ Empirical Risk ♥ ❳ ❘ ♥ ✭ θ ✮ ❂ ✶ ❜ ❵ ✭ θ ❀ Z ✐ ✮ ✿ ♥ ✐ ❂✶ ◮ Empirical risk minimizer ❫ ❜ θ ♥ ❂ ❛r❣ ♠✐♥ ❘ ♥ ✭ θ ✮ ✿ θ ✷ B ❞ ✭ ❘ ✮ Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 3 / 17

  7. Why use non-convex loss? ◮ Comparing to logistic regression, non-convex formulation is robust to ourliers. ◮ This model is the same as neural network with a single layer and a single node. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 4 / 17

  8. Why use non-convex loss? ◮ Comparing to logistic regression, non-convex formulation is robust to ourliers. ◮ This model is the same as neural network with a single layer and a single node. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 4 / 17

  9. ❜ ❘ ♥ ✭ ✮ A negative theoretical result 1996 [AHW ✰ 96]) Theorem (Auer et. al. For the non-convex binary classification problem, for any ♥ and ❞ , ✐ ❂✶ such that the empirical risk ❜ there exists a dataset ✭ x ✐ ❀ ② ✐ ✮ ♥ ❘ ♥ ✭ θ ✮ ❞ ❝ ❞ distinct local minima. has ❜ ♥ Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 5 / 17

  10. A negative theoretical result 1996 [AHW ✰ 96]) Theorem (Auer et. al. For the non-convex binary classification problem, for any ♥ and ❞ , ✐ ❂✶ such that the empirical risk ❜ there exists a dataset ✭ x ✐ ❀ ② ✐ ✮ ♥ ❘ ♥ ✭ θ ✮ ❞ ❝ ❞ distinct local minima. has ❜ ♥ Seems to imply the landscape of the non-convex empirical risk ❜ ❘ ♥ ✭ θ ✮ is very rough. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 5 / 17

  11. A negative theoretical result 1996 [AHW ✰ 96]) Theorem (Auer et. al. For the non-convex binary classification problem, for any ♥ and ❞ , ✐ ❂✶ such that the empirical risk ❜ there exists a dataset ✭ x ✐ ❀ ② ✐ ✮ ♥ ❘ ♥ ✭ θ ✮ ❞ ❝ ❞ distinct local minima. has ❜ ♥ Seems to imply the landscape of the non-convex empirical risk ❜ ❘ ♥ ✭ θ ✮ is very rough. Is this the end of the world of non-convex binary classification? Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 5 / 17

  12. Non-convex formulation of binary classification On real data, we "always" observe a unique minimum! Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 6 / 17

  13. Non-convex formulation of binary classification On real data, we "always" observe a unique minimum! Why? Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 6 / 17

  14. Non-convex formulation of binary classification On real data, we "always" observe a unique minimum! Why? Data generated by nature is not against us! Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 6 / 17

  15. A negative theoretical result Theorem (Auer et. al. . 1996 [AHW ✰ 96]) For the non-convex binary classification problem, for all ♥ ❃ ✵ ✐ ❂✶ such that the empirical risk ❜ there exists a dataset ✭ x ✐ ❀ ② ✐ ✮ ♥ ❘ ♥ ✭ θ ✮ ❞ ❝ ❞ distinct local minima. has ❜ ♥ Seems to imply the landscape of the non-convex empirical risk ❜ ❘ ♥ ✭ θ ✮ is very rough. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 7 / 17

  16. Our main positive result Theorem (Mei, Bai, Montanari. 2016 [MBM16]) Assume X ✐ are i.i.d. sub-Gaussian random vectors, and ❨ ✐ are generated via P ✭ ❨ ✐ ❂ ✶ ❥ X ✐ ✮ ❂ ✛ ✭ ❤ X ✐ ❀ θ ✵ ✐ ✮ . Then there exists a constant ❈ depending on ✍ , such that as long as ♥ ✕ ❈❞ ❧♦❣ ❞ , the following happens with probability at least ✶ � ✍ : ❘ ♥ ✭ θ ✮ has a unique local minimizer ❫ ✭ ❛ ✮ ❜ θ ♥ in B ❞ ✭ 0 ❀ ❘ ✮ . ♣ ✭ ❜ ✮ ❫ θ ♥ satisfies ❦ ❫ θ ♥ � θ ✵ ❦ ✷ ✔ ❈ ✭ ❞ ❧♦❣ ♥ ✮ ❂♥ . ✭ ❝ ✮ Gradient descent converges exponentially fast to ❫ θ ♥ . The landscape of the non-convex empirical risk ❜ ❘ ♥ ✭ θ ✮ is actually smooth! Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 8 / 17

  17. Why assuming a statistical model make the landscape of emprical risk smooth? ✐✿✐✿❞✿ 1 Assuming a statistical model Z ✐ ✘ P Z , ✐ ❂ ✶ ❀ ✿ ✿ ✿ ❀ ♥ , we can define the population risk ✧ ★ ❤ ✐ ♥ ❳ ✶ ❜ ❘ ✭ θ ✮ ❂ E Z ❘ ♥ ✭ θ ✮ ❂ E Z ❵ ✭ θ ❀ Z ✐ ✮ ✿ ♥ ✐ ❂✶ The population risk is usually very smooth. 2 We can transfer the good properties of the population risk to the empirical risk using uniform convergence argument. So empirical risk will be also smooth. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 9 / 17

  18. Why assuming a statistical model make the landscape of emprical risk smooth? ✐✿✐✿❞✿ 1 Assuming a statistical model Z ✐ ✘ P Z , ✐ ❂ ✶ ❀ ✿ ✿ ✿ ❀ ♥ , we can define the population risk ✧ ★ ❤ ✐ ♥ ❳ ✶ ❜ ❘ ✭ θ ✮ ❂ E Z ❘ ♥ ✭ θ ✮ ❂ E Z ❵ ✭ θ ❀ Z ✐ ✮ ✿ ♥ ✐ ❂✶ The population risk is usually very smooth. 2 We can transfer the good properties of the population risk to the empirical risk using uniform convergence argument. So empirical risk will be also smooth. Song Mei (ICME, Stanford) The landscape of empirical risk December 3, 2016 9 / 17

Recommend

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang ,

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang ,

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang , Changyou Chen and Jinhui Xu State University of New York at Bu ff alo International Conference on Machine Learning 2019 Di Wang Non-convex DP-ERM ICML

544 views • 27 slides
The landscape of non-convex losses for statistical learning problems Song Mei Stanford

The landscape of non-convex losses for statistical learning problems Song Mei Stanford

The landscape of non-convex losses for statistical learning problems Song Mei Stanford University October 19, 2017 Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 1 / 32 Deep learning Song Mei

834 views • 66 slides
Introduction to Machine Learning Vapnik Chervonenkis Theory Barnabs Pczos Empirical Risk

Introduction to Machine Learning Vapnik Chervonenkis Theory Barnabs Pczos Empirical Risk

Introduction to Machine Learning Vapnik Chervonenkis Theory Barnabs Pczos Empirical Risk and True Risk 2 Empirical Risk Shorthand: True risk of f (deterministic) : Bayes risk : Let us use the empirical counter part: Empirical risk: 3

1.46k views • 69 slides
Empirical Methods Empirical Methods t= a +b Research Landscape Quantitative =

Empirical Methods Empirical Methods t= a +b Research Landscape Quantitative =

Empirical Methods Empirical Methods t= a +b Research Landscape Quantitative = Positivist/post-positivist approach Evaluate hypotheses via experimentation Qualitative = Constructivist approach Build theory from data Overview:

435 views • 25 slides
Overparametrization for Landscape Design in Non-convex Optimization Jason D. Lee University of

Overparametrization for Landscape Design in Non-convex Optimization Jason D. Lee University of

Overparametrization for Landscape Design in Non-convex Optimization Jason D. Lee University of Southern California October 8, 2018 Jason Lee The State of Non-Convex Optimization Practical observation: Empirically, non-convexity is not an

889 views • 30 slides
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization Zhenxun

Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization Zhenxun

Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization Zhenxun Zhuang 1 , Ashok Cutkosky 2 , Francesco Orabona 1 , 3 1 Department of Computer Science, Boston University 2 Google 3 Department of Electrical &

121 views • 10 slides
Calibration of Convex Surrogate Losses via Property Elicitation Jessie Finocchiaro October 10,

Calibration of Convex Surrogate Losses via Property Elicitation Jessie Finocchiaro October 10,

Calibration of Convex Surrogate Losses via Property Elicitation Jessie Finocchiaro October 10, 2019 Introduction Machine Learning Predictions about Some property of data future events Outline Background Surrogates Calibration

904 views • 44 slides
Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view Perceptron CRF Notation for Linear Models Training data: {(x 1 , y 1 ), (x 2 , y 2 ), , (x N , y N )} Testing data: {(x N+1 , y N+1 ),

418 views • 21 slides
Contents of Presentation Types of losses Causes of losses Prevention of losses

Contents of Presentation Types of losses Causes of losses Prevention of losses

Contents of Presentation Types of losses Causes of losses Prevention of losses Documenting and reporting of losses Types of Losses Marine Internal Inland Internal Loss location: At Port At Warehouse At Food

407 views • 23 slides
CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical

CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical

CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical Risk Minimization Learning: Reminder Known data universe X and an unknown probability distribution D on X Known concept class C and an unknown

341 views • 17 slides
An Empirical Look at the Loss Landscape HEP AI - September 4, 2018 Components of training an

An Empirical Look at the Loss Landscape HEP AI - September 4, 2018 Components of training an

An Empirical Look at the Loss Landscape HEP AI - September 4, 2018 Components of training an image classifier For fixed architecture of ResNet 56 we have: 1. Preprocessing: normalize, shift and flip (show examples) 2. Momentum 3. Weight decay

949 views • 35 slides
Convex hull 1 - 1 Convex hull 1 - 2 Convex hull 1 - 3 Convex hull Definition, extremal

Convex hull 1 - 1 Convex hull 1 - 2 Convex hull 1 - 3 Convex hull Definition, extremal

Convex hull 1 - 1 Convex hull 1 - 2 Convex hull 1 - 3 Convex hull Definition, extremal point Jarvis algorithm Orientation predicate Buggy degenerate example Real RAM model and general position hypothesis Graham

1.41k views • 114 slides
Output of a convex-hull algorithm: a, b, c, g, j A&DS Lecture 16 2 Mary Cryan Polar Angles

Output of a convex-hull algorithm: a, b, c, g, j A&DS Lecture 16 2 Mary Cryan Polar Angles

The Convex Hull Definition 16.1 (1) A set C of points is convex if for all p, q C the whole line segment pq is contained in C . (2) The convex hull of a set S of points is the smallest convex set C that contains S . Observation 16.2 The convex

290 views • 14 slides
ON THE OPTIMIZATION LANDSCAPE OF NEURAL NETWORKS JOAN BRUNA , CIMS + CDS, NYU in collaboration

ON THE OPTIMIZATION LANDSCAPE OF NEURAL NETWORKS JOAN BRUNA , CIMS + CDS, NYU in collaboration

ON THE OPTIMIZATION LANDSCAPE OF NEURAL NETWORKS JOAN BRUNA , CIMS + CDS, NYU in collaboration with D.Freeman (UC Berkeley), Luca Venturi & Afonso Bandeira (NYU) MOTIVATION We consider the standard Empirical Risk Minimization setup: `

683 views • 46 slides
On the Local Minima of the Empirical Risk Chi Jin * 1 , Lydia T. Liu* 1 , Rong Ge 2 , Michael I.

On the Local Minima of the Empirical Risk Chi Jin * 1 , Lydia T. Liu* 1 , Rong Ge 2 , Michael I.

On the Local Minima of the Empirical Risk Chi Jin * 1 , Lydia T. Liu* 1 , Rong Ge 2 , Michael I. Jordan 1 1EECS, University of California, Berkeley. 2Duke University. 1 / 6 Chi Jin On the Local Minima of the Empirical Risk Overview Nonconvex

425 views • 14 slides
Reducing Losses from Flood Risks: Role of the Behavioral Risk Audit Howard Kunreuther

Reducing Losses from Flood Risks: Role of the Behavioral Risk Audit Howard Kunreuther

Reducing Losses from Flood Risks: Role of the Behavioral Risk Audit Howard Kunreuther kunreuth@wharton.upenn.edu James G. Dinan Professor Emeritus of Decision Sciences & Public Policy Co-Director, Risk Management and Decision Processes

345 views • 19 slides
Empirical Power Law for Comment (cont-d) Our Explanation Company Losses: Our Explanation

Empirical Power Law for Comment (cont-d) Our Explanation Company Losses: Our Explanation

Formulation of the . . . Formulation of the . . . Comment Empirical Power Law for Comment (cont-d) Our Explanation Company Losses: Our Explanation (cont-d) Our Explanation (cont-d) A Probability-Based References Explanation Home Page

427 views • 9 slides
3.1 Online Convex Programming Definition 3.1.1 (Convex Set) A set of vectors X R n is convex

3.1 Online Convex Programming Definition 3.1.1 (Convex Set) A set of vectors X R n is convex

CS/CNS/EE 253: Advanced Topics in Machine Learning Topic: Online Gradient Descent Lecturer: Daniel Golovin Scribe: Esther Wang Date: Jan. 11, 2010 3.1 Online Convex Programming Definition 3.1.1 (Convex Set) A set of vectors X R n is convex if

93 views • 5 slides
The Empirical Landscape of Trade Policy Chad P. Bown Meredith A. Crowley Peterson Institute

The Empirical Landscape of Trade Policy Chad P. Bown Meredith A. Crowley Peterson Institute

The Empirical Landscape of Trade Policy Chad P. Bown Meredith A. Crowley Peterson Institute & CEPR University of Cambridge forthcoming in Kyle Bagwell and Robert W. Staiger, Eds. The Handbook of Commercial Policy. Netherlands: Elsevier.

595 views • 55 slides
Convex Hell 362 dnc CS 16: Convex Hull Whoops, I mean... Convex Hull Whats a Convex Hull?

Convex Hell 362 dnc CS 16: Convex Hull Whoops, I mean... Convex Hull Whats a Convex Hull?

CS 16: Convex Hull Welcome to... Convex Hell 362 dnc CS 16: Convex Hull Whoops, I mean... Convex Hull Whats a Convex Hull? 363 dnc CS 16: Convex Hull What is the Convex Hull? Let S be a set of points in the plane. Intuition: Imagine

720 views • 19 slides
The Empirical Landscape of Trade Policy Chad P. Bown and Meredith A. Crowley World Bank

The Empirical Landscape of Trade Policy Chad P. Bown and Meredith A. Crowley World Bank

The Empirical Landscape of Trade Policy Chad P. Bown and Meredith A. Crowley World Bank & CEPR and Cambridge May 2015 CPB & MAC (WB, CEPR & Cambridge ) The Empirical Landscape of Trade Policy May 2015 1 / 44 What do

1.07k views • 44 slides
Non Convex Minimization using Convex Relaxation Some Hints to Formulate Equivalent Convex Energies

Non Convex Minimization using Convex Relaxation Some Hints to Formulate Equivalent Convex Energies

Non Convex Minimization using Convex Relaxation Some Hints to Formulate Equivalent Convex Energies Mila Nikolova ( CMLA, ENS Cachan, CNRS, France ) SIAM Imaging Conference (IS14) Hong Kong Minitutorial: May 13, 2014 Outline 1. Energy

455 views • 26 slides
Where the disaster risk technologies are taking us? By Luis Ceferino Risk Post-EQ Risk

Where the disaster risk technologies are taking us? By Luis Ceferino Risk Post-EQ Risk

Risk analysis beyond insurance. Where the disaster risk technologies are taking us? By Luis Ceferino Risk Post-EQ Risk Insurance tools financial losses sector data The new generation of risk tools. How do they fit into the risk

184 views • 17 slides
14. Convex programming Convex sets and functions Convex programs Hierarchy of

14. Convex programming Convex sets and functions Convex programs Hierarchy of

CS/ECE/ISyE 524 Introduction to Optimization Spring 201718 14. Convex programming Convex sets and functions Convex programs Hierarchy of complexity Example: geometric programming Laurent Lessard (www.laurentlessard.com)

400 views • 24 slides

More recommend