the effect of network width on stochastic gradient
play

The Effect of Network Width on Stochastic Gradient Descent and - PowerPoint PPT Presentation

Nov 20, 2022 •275 likes •377 views

The Effect of Network Width on Stochastic Gradient Descent and Generalization Daniel S. Park Google ICML 2019 Daniel S. Park (Google) ICML 2019 1 / 9 Work with Jascha Sohl-Dickstein, Quoc V. Le and Samuel L. Smith. Daniel S. Park (Google)

  1. The Effect of Network Width on Stochastic Gradient Descent and Generalization Daniel S. Park Google ICML 2019 Daniel S. Park (Google) ICML 2019 1 / 9

  2. Work with Jascha Sohl-Dickstein, Quoc V. Le and Samuel L. Smith. Daniel S. Park (Google) ICML 2019 2 / 9

  3. Motivation Let us assume that • we found hyperparameters that maximize test set accuracy for a given network, • but now we want to make the network bigger by widening all the channels by factor w . What do we do with the hyperparameters for the new network? Daniel S. Park (Google) ICML 2019 3 / 9

  4. Main Result We find a rule that governs how hyperparameters that maximize test accuracy change when the network width is varied. The rule is that the optimal value of the normalized noise scale (which is a function of the hyperparameters of SGD) scales proportionally to the width of the network. Daniel S. Park (Google) ICML 2019 4 / 9

  5. The Normalized Noise Scale ¯ g 1 • ¯ g = ǫ init governs how noisy the SGD is. B (1 − m ) · σ 2 • ¯ g determines the generalization performance. ∗ *Mandt et al. (2017); Chaudhari & Soatto (2017); Jastrzebski et al. (2017); Smith & Le (2017). Daniel S. Park (Google) ICML 2019 5 / 9

  6. Rule for Hyperparameter Selection • There exists a simple rule for hyperparameter selection: Increase ¯ g proportionally with w . Daniel S. Park (Google) ICML 2019 6 / 9

  7. Wider networks require smaller batch sizes • To maximize generalization performance, wide networks (eventually) need to be trained with small batch sizes: B opt ≤ (constant) w Daniel S. Park (Google) ICML 2019 7 / 9

  8. Bigger networks perform better due to noise resistance • Bigger networks have better peak test set performance which is reached at higher noise scales. Daniel S. Park (Google) ICML 2019 8 / 9

  9. Visit our poster (Pacific Ballroom #55) to learn more. Thank you! Daniel S. Park (Google) ICML 2019 9 / 9

Recommend

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate Jingfeng Wu , Difan Zou, Vladimir Braverman, Quanquan Gu Johns Hopkins University & UCLA November 2020 Overview

193 views • 16 slides
Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides
Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent, Momentum, weight decay New: (last 5-6 years) Dropout ReLUs Batch Normalization Reminder: Overfitting, in images Classification just right

1.61k views • 132 slides
Some References P. Carpentier Master MMMEF Cours MNOS 2014-2015 263 / 263 Stochastic

Some References P. Carpentier Master MMMEF Cours MNOS 2014-2015 263 / 263 Stochastic

Stochastic Approximation and Stochastic Gradient Optimization Theory and Large Scale Systems Dual Effect and Discretization Stochastic Programming and Stochastic Optimal Control Some References P. Carpentier Master MMMEF Cours MNOS

385 views • 9 slides
Overview of the Stochastic Gradient Method December 02, 2020 P. Carpentier Master Optimization

Overview of the Stochastic Gradient Method December 02, 2020 P. Carpentier Master Optimization

Stochastic Gradient Algorithm Connexion with Stochastic Approximation Asymptotic Efficiency and Averaging Practical Considerations Overview of the Stochastic Gradient Method December 02, 2020 P. Carpentier Master Optimization Stochastic

338 views • 33 slides
Applications of the Stochastic Gradient Method December 11, 2019 P. Carpentier Master

Applications of the Stochastic Gradient Method December 11, 2019 P. Carpentier Master

Two Exercices about Stochastic Gradient Option Pricing Problem and Variance Reduction Spatial Rendez-vous Under Probability Constraint Applications of the Stochastic Gradient Method December 11, 2019 P. Carpentier Master Optimization

819 views • 55 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS 2020 Aaron Mishkin, amishkin@cs.ubc.ca 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of

634 views • 21 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS 2019 Aaron Mishkin 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of the main

1.22k views • 21 slides
Adaptive primal-dual stochastic gradient methods Yangyang Xu Mathematical Sciences, Rensselaer

Adaptive primal-dual stochastic gradient methods Yangyang Xu Mathematical Sciences, Rensselaer

Adaptive primal-dual stochastic gradient methods Yangyang Xu Mathematical Sciences, Rensselaer Polytechnic Institute October 26, 2019 1 / 22 Stochastic gradient method stochastic program: F ( x ; ) min x X f ( x ) = E N

959 views • 22 slides
Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner New requirement for the final project: For the first time ever, researchers who submit

619 views • 34 slides
On the steplength selection in Stochastic Gradient Methods Giorgia Franchini

On the steplength selection in Stochastic Gradient Methods Giorgia Franchini

Introduction Stochastic Gradient Methods and their properties A numerical experiment: the test problem Future developments On the steplength selection in Stochastic Gradient Methods Giorgia Franchini giorgia.franchini@unimore.it Universit

528 views • 19 slides
Exponential convergence of testing error for stochastic gradient methods Loucas Pillaud-Vivien

Exponential convergence of testing error for stochastic gradient methods Loucas Pillaud-Vivien

Exponential convergence of testing error for stochastic gradient methods Loucas Pillaud-Vivien INRIA - Ecole Normale Sup erieure, Paris, France Joint work with Alessandro Rudi and Francis Bach COLT - July 2018 1/10 Stochastic Gradient

376 views • 20 slides
Project: More Experiments on Stochastic Gradient Methods Last updated: May 25, 2020 May 25, 2020

Project: More Experiments on Stochastic Gradient Methods Last updated: May 25, 2020 May 25, 2020

Project: More Experiments on Stochastic Gradient Methods Last updated: May 25, 2020 May 25, 2020 1 / 17 Goal We want to know more the internal details of simpleNN We want to roughly compare the two stochastic gradient approaches: SG with

502 views • 17 slides
Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours

Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours MNOS

702 views • 49 slides
Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background

Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background

Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background In many practical applications, the objective function is a large sum: N f ( x ) = i = 1 f i ( x ) Issues and questions: Evaluating

381 views • 9 slides
Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade 2018 c University of Washington cse446-staff@cs.washington.edu 1 / 12 Announcements Midterm: Weds, Feb 7th. Policies: You may use a single

333 views • 18 slides
Block stochastic gradient update method Yangyang Xu and Wotao Yin IMA, University of

Block stochastic gradient update method Yangyang Xu and Wotao Yin IMA, University of

Block stochastic gradient update method Yangyang Xu and Wotao Yin IMA, University of Minnesota Department of Mathematics, UCLA November 1, 2015 This work was done while in Rice University 1 / 26 Stochastic gradient method

680 views • 26 slides
The Effect of Planarization on Width David Eppstein 25th International Symposium on Graph

The Effect of Planarization on Width David Eppstein 25th International Symposium on Graph

The Effect of Planarization on Width David Eppstein 25th International Symposium on Graph Drawing & Network Visualization Boston, Massachusetts, September 2017 Planarization Draw a graph with simple crossings (two edges/crossing) Then,

158 views • 13 slides
Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization

Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization

Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization Vien V. Mai and Mikael Johansson KTH - Royal Institute of Technology Stochastic optimization Stochastic optimization problem: minimize f ( x

381 views • 20 slides
Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla December 17, 2011 Peter J. Haas Yannis Sismanis Christina Teflioudi Faraz Makari Outline Matrix Factorization Stochastic Gradient Descent

1.24k views • 75 slides
Effect of Spike-Timing-Dependent Plasticity on Stochastic Spike Synchronization in A Small-World

Effect of Spike-Timing-Dependent Plasticity on Stochastic Spike Synchronization in A Small-World

Effect of Spike-Timing-Dependent Plasticity on Stochastic Spike Synchronization in A Small-World Neuronal Network S.-Y. Kim and W. Lim Institute for Computational Neuroscience Daegu National University of Education Stochastic Spike

264 views • 9 slides
Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla August 23, 2011 Peter J. Haas Yannis Sismanis Erik Nijkamp Outline Matrix Factorization Stochastic Gradient Descent Distributed SGD with MapReduce

1.05k views • 81 slides
Stochastic Gradient Push for Distributed Deep Learning Mido Assran, Nicolas Loizou, Nicolas

Stochastic Gradient Push for Distributed Deep Learning Mido Assran, Nicolas Loizou, Nicolas

Stochastic Gradient Push for Distributed Deep Learning Mido Assran, Nicolas Loizou, Nicolas Ballas, Mike Rabbat 1 Data Parallel Training parallel Stochastic Gradient Descent x , f 2 ( x ) x , f 1 ( x ) ( f i ( x ) ) n 1 x

915 views • 13 slides
Setting Adaptativity of Stochastic Gradient Descent Aymeric Dieuleveut F. Bach, Non parametric

Setting Adaptativity of Stochastic Gradient Descent Aymeric Dieuleveut F. Bach, Non parametric

Setting Adaptativity of Stochastic Gradient Descent Aymeric Dieuleveut F. Bach, Non parametric stochastic approximation with large step sizes , in the Annals of Statistics Setting : random-design least-squares regression problem in a RKHS

276 views • 3 slides

More recommend