Gradient Descent Finds Global Minima of Deep Neural Networks Simon - PowerPoint PPT Presentation

Dec 06, 2023 •178 likes •264 views

Gradient Descent Finds Global Minima of Deep Neural Networks Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, Xiyu Zhai 1 Empirical Observations on Empirical Risk Zhang et al, 2017, Understanding Deep Learning Requires Rethinking

Gradient Descent Finds Global Minima of Deep Neural Networks Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, Xiyu Zhai 1
Empirical Observations on Empirical Risk • Zhang et al, 2017, Understanding Deep Learning Requires Rethinking Generalization. Randomization Test: replace true labels by random labels. Observations: Empirical Risk-> 0 for both true labels and random labels. Conjecture: because neural networks are over-parameterized. Open Problem: why gradient descent can find a neural network that fits all labels. 2
Setup { x i , y i } n • Training Data: i =1 , x i ∈ R d , y i ∈ R • A Model. • Fully connected neural network: f ( θ , x ) = W L σ ( W L − 1 · · · W 2 σ ( W 1 x ) · · · ) • A loss function. n • Quadratic loss: R ( θ ) = 1 X ( f ( θ , x i ) − y i ) 2 2 n i =1 • An optimization algorithm: • Gradient descent: θ ( t + 1) ← θ ( t ) − η ∂ R ( θ ( t )) ∂θ ( t ) 3
Trajectory-based Analysis θ ( t + 1) ← θ ( t ) − η ∂ R ( θ ( t )) ∂θ ( t ) • Trajectory of parameters: θ (0) , θ (1) , θ (2) , · · · • Predictions: u i ( t ) , f ( θ ( t ) , x i ) , u ( t ) , ( u 1 ( t ) , . . . , u n ( t )) > ∈ R n • Trajectory of predictions: u (0) , u (1) , u (2) , . . . 4
Proof Sketch • Simplified form (continuous time): L du ( t ) ij ( t ) = 1 n h ∂ u i ( t ) ∂ W ` ( t ) , ∂ u j ( t ) X H ` ( t ) ( y − u ( t )) H ` = − ∂ W ` ( t ) i dt ` =1 • Random initialization + concentration + perturbation analysis: L L L X X X H ` (0) → H ∞ H ` ( t ) → H ` (0) , ∀ t ≥ 0 lim lim m →∞ m →∞ ` =1 ` =1 ` =1 • Linear ODE theory: k u ( t ) � y k 2 2  exp ( � λ 0 t ) k u (0) � y k 2 2 , λ 0 = λ min ( H ∞ ) 5
Main Results Theorem 1: For fully-connected neural network with smooth activation, if ! = poly ', 2 * , 1/- . and step 1 2 size / = 0 3 4 5 6(8) , then with high probability over random initialization we have: for : = 1,2, … @ <(=(0)) . < = : ≤ 1 − /- . First global linear convergence guarantee for deep NN. • Exponential dependence due to error propagation. • 6
Main Results (Cont’d) Theorem 2: For ResNet or Convolutional ResNet with smooth activation, if ! = 0 1 poly ', ), 1/, - and step size . = / 2 3 , then with high probability over random initialization we have: for 4 = 1,2, … ; 7(8(0)) . 7 8 4 ≤ 1 − ., - ResNet architecture makes the error propagation more stable => • exponential improvement over fully-connected neural networks. 7
Learn more @ Pacific Ball Room #80 8

Recommend

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit: Vlad Morariu Training (Deep) Neural Networks Computational graphs Improvements to gradient descent Stochastic gradient descent Momentum

814 views • 21 slides

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner New requirement for the final project: For the first time ever, researchers who submit

619 views • 34 slides

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao and Quanquan Gu Computer Science Department 1 / 14 Learning Over-parameterized DNNs Empirical observation on extremely wide deep neural networks

631 views • 14 slides

Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari

Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari UCLA VISION LAB 1 The question measures disagreement of predictions with ground truth x = argmin f ( x ) Cat Dog ... x weights aka

496 views • 25 slides

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects Zhanxing Zhu , Jingfeng Wu , Bing Yu, Lei Wu, Jinwen Ma. Peking University Beijing Institute of Big Data Research

319 views • 15 slides

CS7015 (Deep Learning) : Lecture 3 Sigmoid Neurons, Gradient Descent, Feedforward Neural Networks,

CS7015 (Deep Learning) : Lecture 3 Sigmoid Neurons, Gradient Descent, Feedforward Neural Networks, Representation Power of Feedforward Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of

869 views • 70 slides

On Gradient Descent and Local vs. Global Optimum We conjecture that both simulating anneal- ing

On Gradient Descent and Local vs. Global Optimum We conjecture that both simulating anneal- ing and SGD converge to the band of low criticial points, and that all criticial points found are local minima of high quality measured by the

368 views • 26 slides

Gradient Descent and the Structure of Neural Network Cost Functions presentation by Ian

Gradient Descent and the Structure of Neural Network Cost Functions presentation by Ian Goodfellow adapted for www.deeplearningbook.org from a presentation to the CIFAR Deep Learning summer school on August 9, 2015 Optimization -Exhaustive

514 views • 38 slides

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides

Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of theoretical studies of deep learning 2. Optimization theory of deep neural networks 1) Gradient finds global optima 2) Gram-Gauss-Newton Algorithm

484 views • 31 slides

Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob

Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob CPSC 533 Winter 2001 Learning by Gradient Descent Definition of the Learning Problem Let us start with the simple case of linear cells, which

558 views • 22 slides

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent, Momentum, weight decay New: (last 5-6 years) Dropout ReLUs Batch Normalization Reminder: Overfitting, in images Classification just right

1.61k views • 132 slides

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides

Artificial Neural Networks Threshold units Gradient descent Multilayer networks

Artificial Neural Networks Threshold units Gradient descent Multilayer networks Backpropagation Hidden layer representations Example: Face Recognition Advanced topics 1 Connectionist Models Consider humans: Neuron

535 views • 32 slides

1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

MAP562 Optimal design of structures by Gr egoire Allaire, Thomas Wick Ecole Polytechnique Academic year 2016-2017 Gradient descent and Newtons method for minimization in a PDE / function space context We recapitulate the basic steps to

327 views • 5 slides

Neural Networks: Computation + Gradient Descent LING572 Advanced Statistical Methods in NLP

Neural Networks: Computation + Gradient Descent LING572 Advanced Statistical Methods in NLP February 27 2020 1 Todays Outline Computation: the forward pass Functional form / matrix notation Parameters and Hyperparameters

832 views • 40 slides

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade 2018 c University of Washington cse446-staff@cs.washington.edu 1 / 12 Announcements Midterm: Weds, Feb 7th. Policies: You may use a single

333 views • 18 slides

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on Mathematical Theory of Deep Neural Networks October 31, 2019 New York City, NY Need for New Optimization Theory The mystery of deep neural networks

549 views • 17 slides

CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University of Virginia Overview 1. Gradient Descent 2. Stochastic Gradient Descent 3. SGD with Momentum 4. Adaptive Learning Rates 1 Gradient Descent

767 views • 66 slides

LOGISTIC REGRESSION, GRADIENT LOGISTIC REGRESSION, GRADIENT DESCENT, NEWTON DESCENT, NEWTON

LOGISTIC REGRESSION, GRADIENT LOGISTIC REGRESSION, GRADIENT DESCENT, NEWTON DESCENT, NEWTON Matthieu R Bloch Thursday, January 30, 2020 1 LOGISTICS LOGISTICS TAs and Office hours Monday: Mehrdad (TSRB 523a) - 2pm-3:15pm Tuesday: TJ (VL

299 views • 14 slides

CS7015 (Deep Learning) : Lecture 5 Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated

CS7015 (Deep Learning) : Lecture 5 Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated GD, Stochastic GD, AdaGrad, RMSProp, Adam Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1

1.24k views • 94 slides

Gradient Descent Michail Michailidis & Patrick Maiden Outline

Gradient Descent Michail Michailidis & Patrick Maiden Outline Mo4va4on Gradient Descent Algorithm Issues & Alterna4ves Stochas4c Gradient Descent

840 views • 29 slides

Neural Networks Module2 : learning with Gradient Descent module 2: numerical optimization

Neural Networks Module2 : learning with Gradient Descent module 2: numerical optimization LEARNING PERFORMANCE REPRESENTATION DATA PROBLEM EVALUATION RAW DATA CLUSTERING FEATURES housing data train/test error, accuracy spam data Cross

639 views • 19 slides