The landscape of non-convex losses for statistical learning problems - PowerPoint PPT Presentation

The landscape of non-convex losses for statistical learning problems Song Mei Stanford University October 19, 2017 Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 1 / 32

Deep learning Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 2 / 32

Convolutional Neural Network Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 3 / 32

Non-convex optimization Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 4 / 32

Why does non-convex neural network perform well? Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 5 / 32

Why does some non-convex optimization perform well? Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 6 / 32

Why does some non-convex optimization perform well? ◮ Stochastic gradient descent escape bad local minima. ◮ Good initialization escape bad local minima. ◮ Globally there are less bad local minima. ◮ .... Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 6 / 32

Non-convex optimization: analysis of global geometry Number and locations of saddle points and local minima. Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 7 / 32

Let’s do it! The objective function ♥ ❳ ✶ ❢ ② ✐ � ✛ ✭ ❲ ❦ ✁ ✁ ✁ ✛ ✭ ❲ ✷ ✁ ✛ ✭ ❲ ✶ ① ✐ ✮✮✮ ❣ ✷ ♠✐♥ ♥ ❲ ✐ ✐ ❂✶ Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 8 / 32

Let’s do it! The objective function ♥ ❳ ✶ ❢ ② ✐ � ✛ ✭ ❲ ✷ ✁ ✛ ✭ ❲ ✶ ① ✐ ✮✮ ❣ ✷ ♠✐♥ ♥ ❲ ✐ ✐ ❂✶ Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 8 / 32

Let’s do it! The objective function ♥ ❳ ✶ ❢ ② ✐ � ✛ ✭ ❤ ✒❀ ① ✐ ✐ ✮ ❣ ✷ ♠✐♥ ♥ ✒ ✐ ❂✶ Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 8 / 32

Binary linear classification The model ③ ✐ ❂ ✭ ① ✐ ❀ ② ✐ ✮ . ① ✐ ✷ R ❞ , ② ✐ ✷ ❢ ✵ ❀ ✶ ❣ . Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 9 / 32

One node neural network The model ③ ✐ ❂ ✭ ① ✐ ❀ ② ✐ ✮ . ① ✐ ✷ R ❞ , ② ✐ ✷ ❢ ✵ ❀ ✶ ❣ . ◮ Convex logit loss ( ❵ ❝ is cvx in ✒ ) ❵ ❝ ✭ ✒ ❀ ③ ✮ ❂ ② ❤ ①❀ ✒ ✐ � ❧♦❣ ❢ ✶ ✰ ❡①♣✭ ❤ ①❀ ✒ ✐ ✮ ❣ ✿ ◮ Non-convex loss ( ❵ is not cvx in ✒ ) ❵ ✭ ✒ ❀ ③ ✮ ❂ ❢ ② � ✛ ✭ ❤ ①❀ ✒ ✐ ✮ ❣ ✷ ❀ where ✛ ✭ t ✮ ❂ ✶ ❂ ✭✶ ✰ ❡①♣✭ t ✮✮ ✿ ◮ Empirical Risk ♥ ♥ ❳ ❳ ❘ ♥ ✭ ✒ ✮ ❂ ✶ ❵ ✭ ✒ ❀ ③ ✐ ✮ ❂ ✶ ❜ ❢ ② ✐ � ✛ ✭ ❤ ✒❀ ① ✐ ✐ ✮ ❣ ✷ ✿ ♥ ♥ ✐ ❂✶ ✐ ❂✶ ◮ Empirical risk minimizer ❫ ❜ ✒ ♥ ❂ ❛r❣ ♠✐♥ ❘ ♥ ✭ ✒ ✮ ✿ ✒ ✷ B ❞ ✭ ❘ ✮ Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 10 / 32

❜ ❘ ♥ ✭ ✒ ✮ A negative theoretical result Theorem (Auer et. al. . 1996) For the one node neural network, ✽ ♥❀ ❞ ❃ ✵ , there exists a dataset ❞ ❝ ❞ distinct local ✭ ① ✐ ❀ ② ✐ ✮ ♥ ✐ ❂✶ such that the empirical risk ❜ ❘ ♥ ✭ ✒ ✮ has ❜ ♥ minima. Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 11 / 32

A negative theoretical result Theorem (Auer et. al. . 1996) For the one node neural network, ✽ ♥❀ ❞ ❃ ✵ , there exists a dataset ❞ ❝ ❞ distinct local ✭ ① ✐ ❀ ② ✐ ✮ ♥ ✐ ❂✶ such that the empirical risk ❜ ❘ ♥ ✭ ✒ ✮ has ❜ ♥ minima. The landscape of ❜ ❘ ♥ ✭ ✒ ✮ is very rough. Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 11 / 32

A negative theoretical result Theorem (Auer et. al. . 1996) For the one node neural network, ✽ ♥❀ ❞ ❃ ✵ , there exists a dataset ❞ ❝ ❞ distinct local ✭ ① ✐ ❀ ② ✐ ✮ ♥ ✐ ❂✶ such that the empirical risk ❜ ❘ ♥ ✭ ✒ ✮ has ❜ ♥ minima. The landscape of ❜ ❘ ♥ ✭ ✒ ✮ is very rough. Is this the end of the world of deep learning? Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 11 / 32

Real data experiment ◮ The "Australian" data set from Statlog: ❞ ❂ ✶✶ , ♥ ❂ ✻✽✸ . ◮ Random initialization ✒ ✭✵✮ ✘ ◆ ✭ 0 ❀ ■ ❞ ✮ . ◮ Run gradient descent and track the path ✒ ✭ ❦ ✮ . ◮ Generate multiple paths with independent initializations. ◮ Plot standard deviation over paths st❞✭ ✒ ✭ ❦ ✮✮ versus ❦ . Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 12 / 32

The landscape of non-convex losses for statistical learning problems - PowerPoint PPT Presentation

The landscape of non-convex losses for statistical learning problems Song Mei Stanford University October 19, 2017 Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 1 / 32 Deep learning Song Mei

The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016

Overparametrization for Landscape Design in Non-convex Optimization Jason D. Lee University of

Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization Zhenxun

Calibration of Convex Surrogate Losses via Property Elicitation Jessie Finocchiaro October 10,

Contents of Presentation Types of losses Causes of losses Prevention of losses

Convex hull 1 - 1 Convex hull 1 - 2 Convex hull 1 - 3 Convex hull Definition, extremal

Output of a convex-hull algorithm: a, b, c, g, j A&DS Lecture 16 2 Mary Cryan Polar Angles

3.1 Online Convex Programming Definition 3.1.1 (Convex Set) A set of vectors X R n is convex

Convex Hell 362 dnc CS 16: Convex Hull Whoops, I mean... Convex Hull Whats a Convex Hull?

Non Convex Minimization using Convex Relaxation Some Hints to Formulate Equivalent Convex Energies

14. Convex programming Convex sets and functions Convex programs Hierarchy of

Convex Analysis Jos e De Don a September 2004 Centre of Complex Dynamic Systems and

A glimpse into convex geometry 5 \ A glimpse into convex geometry Two

Convex hulls of spheres and convex hulls of convex polytopes lying on parallel hyperplanes

Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex

Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex

Minimizing within convex bodies using a convex hull method Edouard Oudet Thomas

Convex Calibrated Surrogates for Low-Rank Loss Matrices with Applications to Subset Ranking

CS675: Convex and Combinatorial Optimization Spring 2018 Convex Optimization Problems

16. Review of convex optimization Convex sets and functions Convex programming models

CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions Instructor: Shaddin

CS675: Convex and Combinatorial Optimization Fall 2019 Convex Functions Instructor: Shaddin

Overall Equipment Effectiveness OEE Calculation Eight Major Plant Losses Sr. No. Losses

CS133 Computational Geometry Convex Hull 1 Convex Hull Given a set of n points, find the

The landscape of non-convex losses for statistical learning problems - PowerPoint PPT Presentation

The landscape of non-convex losses for statistical learning problems Song Mei Stanford University October 19, 2017 Song Mei (Stanford University) The landscape of non-convex optimization October 19, 2017 1 / 32 Deep learning Song Mei

The landscape of empirical risk for non-convex losses Song Mei ICME, Stanford December 3, 2016

Overparametrization for Landscape Design in Non-convex Optimization Jason D. Lee University of

Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization Zhenxun

Calibration of Convex Surrogate Losses via Property Elicitation Jessie Finocchiaro October 10,

Contents of Presentation Types of losses Causes of losses Prevention of losses

Convex hull 1 - 1 Convex hull 1 - 2 Convex hull 1 - 3 Convex hull Definition, extremal

Output of a convex-hull algorithm: a, b, c, g, j A&amp;DS Lecture 16 2 Mary Cryan Polar Angles

3.1 Online Convex Programming Definition 3.1.1 (Convex Set) A set of vectors X R n is convex

Convex Hell 362 dnc CS 16: Convex Hull Whoops, I mean... Convex Hull Whats a Convex Hull?

Non Convex Minimization using Convex Relaxation Some Hints to Formulate Equivalent Convex Energies

14. Convex programming Convex sets and functions Convex programs Hierarchy of

Convex Analysis Jos e De Don a September 2004 Centre of Complex Dynamic Systems and

A glimpse into convex geometry 5 \ A glimpse into convex geometry Two

Convex hulls of spheres and convex hulls of convex polytopes lying on parallel hyperplanes

Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex

Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex

Minimizing within convex bodies using a convex hull method Edouard Oudet Thomas

Convex Calibrated Surrogates for Low-Rank Loss Matrices with Applications to Subset Ranking

CS675: Convex and Combinatorial Optimization Spring 2018 Convex Optimization Problems

16. Review of convex optimization Convex sets and functions Convex programming models

CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions Instructor: Shaddin

CS675: Convex and Combinatorial Optimization Fall 2019 Convex Functions Instructor: Shaddin

Overall Equipment Effectiveness OEE Calculation Eight Major Plant Losses Sr. No. Losses

CS133 Computational Geometry Convex Hull 1 Convex Hull Given a set of n points, find the

Output of a convex-hull algorithm: a, b, c, g, j A&DS Lecture 16 2 Mary Cryan Polar Angles