stochastic gradient descent
play

Stochastic Gradient Descent Many slides attributable to: Prof. - PowerPoint PPT Presentation

Jul 12, 2023 •41 likes •241 views

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/ Stochastic Gradient Descent Many slides attributable to: Prof. Mike Hughes Erik Sudderth (UCI), Emily Fox (UW), Finale Doshi-Velez (Harvard) James,

  1. Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/ Stochastic Gradient Descent Many slides attributable to: Prof. Mike Hughes Erik Sudderth (UCI), Emily Fox (UW), Finale Doshi-Velez (Harvard) James, Witten, Hastie, Tibshirani (ISL/ESL books) 2

  2. Objectives Today (day 12) Stochastic Gradient Descent • Review: Gradient Descent • Repeatedly step downhill until converged • Review: Training Neural Nets with Backprop • Backprop = chain rule plus dynamic programming • L-BFGS : How to step in better direction? • Stochastic Gradient Descent : How to go fast? Mike Hughes - Tufts COMP 135 - Fall 2020 3

  3. <latexit sha1_base64="QD3y6jOAWeQ5sqrhpmn6rqR5jJ8=">ACN3icbVBNSyNBEO3xY9Wsu0Y9emkMC3sKM+7CehS9iAdRMCqkQ6jp1GQaez7orhHCmH/lxb/hTS8eFPHqP7AnieLHPmjq8V4V1fXCXCtLvn/jTU3PzH6bm1+ofV/8XOpvrxybLPCSGzJTGfmNASLWqXYIkUaT3ODkIQaT8Kznco/OUdjVZYe0SDHTgL9VEVKAjmpW98XoPMYuJC9jLjQGNEFf60iMiBLkYMhBXr4xrigGAmGfI8Lo/px1Tqu3XrDb/oj8K8kmJAGm+CgW78WvUwWCaYkNVjbDvycOmW1R2oc1kRhMQd5Bn1sO5pCgrZTju4e8l9O6fEoM+6lxEfq+4kSEmsHSeg6E6DYfvYq8X9eu6Bos1OqNC8IUzleFBWaU8arEHlPGZSkB46ANMr9lcsYXFjkoq65EILPJ38lxvN4E9z4/BvY2t7Esc8W2Pr7DcL2D+2xXbZAWsxyS7ZLbtnD96Vd+c9ek/j1ilvMrPKPsB7fgFLZq1r</latexit> Review: Gradient Descent in 1D input: initial θ ∈ R input: step size α ∈ R + while not converged: θ ← θ − α d d θ J ( θ ) Q: How far to step in that direction? Q: Which direction to step? � � � � ∂ � � � � A: α · ∂θ J A: Straight downhill � � � � � � � � (steepest descent at current location) Step size parameter picked in advance, unaware of current location Mike Hughes - Tufts COMP 135 - Fall 2020 4

  4. <latexit sha1_base64="YmhBdBy1tgcD8mJGsWOwVKwghNI=">ACLnicbVBNa9tAEF2lH3GVfjNsZelJuBejOQUkqNJKJSeHKgTg2XMaD2yFq9WYncUMIp/US75K80h0IbSa35G1x+F1u7AMo/3jA7Ly6UtBQE372dJ0+fPd+tvfD3Xr56/a+/bC5qUR2BO5yk0/BotKauyRJIX9wiBkscLeHq20C+v0FiZ680K3CYwUTLRAogR43qnyJQRQo8EuOceKQwoes/zY80xApGVUQpEsz5l+YKfAjIyfpwrnqo3ojaAXL4tsgXIMGW1d3VL+LxrkoM9QkFg7CIOChUYkLh3I9KiwWIKUxw4KCGDO2wWp4754eOGfMkN+5p4kv274kKMmtnWeycGVBqN7UF+T9tUFJyMqykLkpCLVaLklJxyvkiOz6WBgWpmQMgjHR/5SIFA4Jcwr4LIdw8eRtctFvhUat9/rHROV3HUWPv2HvWZCE7Zh32mXVZjwl2w76xH+zBu/XuvZ/er5V1x1vPHLB/ynv8DVt1qL8=</latexit> <latexit sha1_base64="N3+JepYHD0brMFSlWy+mt6ByhW4=">ACfnichVHLatwFJXdV+q+ps2yG7VDStqSqZ0Umk0hNJuQVQqdJDAy5lpzPSMiy0a6DgzGn9Ef6y7f0k01M6bkUegFweHcx86N6+1chTHV0F47/6Dh482HkdPnj57/mLw8tWpqxorcSwrXdnzHBxqZXBMijSe1xahzDWe5ReHy/zZJVqnKvODFjWmJcyMKpQE8lQ2+CkM5BqyVtAcCTp+vL1G7/nXSGgsaBKJHGfKtGAtLpWdpEoLMhW1GBJge7+Ir4uzWLfJhKC/1eYrIRopn3zSFg1m1OaDYbxKF4FvwuSHgxZHyfZ4JeYVrIp0ZDU4NwkiWtK2+U0qdH3bRzWIC9ghMPDZTo0nZlX8e3PDPlRWX9M8RX7PWKFkrnFmXulSXQ3N3OLcl/5SYNFftpq0zdEBq5HlQ0mlPFl7fgU2VRkl54ANIqvyuXc/CWkb9Y5E1Ibn/5LjdHSV7o93vn4cH3o7Nthr9pZts4R9YQfsiJ2wMZPsd/Am+B8DFn4LtwJP62lYdDXbLIbEe7/AYrCwhU=</latexit> <latexit sha1_base64="1JLgwDRjFbF3NXnXhXd7r/jcJ0g=">ACH3icbZBNSwMxEIazflu/qh69BIugB8tuFfUoehFPClaFbimz6bQNZrNLMiuUpf/Ei3/FiwdFxJv/xrTdg18vBJ68M0Myb5Qqacn3P72Jyanpmdm5+dLC4tLySnl17domRFYF4lKzG0EFpXUWCdJCm9TgxBHCm+iu9Nh/eYejZWJvqJ+is0Yulp2pAByVqt8EFIPCXjYRbK8uOzyEFTac6GSErH/sDfr49p1WueJX/ZH4XwgKqLBCF63yR9hORBajJqHA2kbgp9TMwZAUCgelMLOYgriDLjYcaojRNvPRfgO+5Zw27yTGHU185H6fyCG2th9HrjMG6tnftaH5X62RUeomUudZoRajB/qZIpTwodh8bY0KEj1HYAw0v2Vix4YEOQiLbkQgt8r/4XrWjXYq9Yu9yvHJ0Uc2yDbJtFrBDdszO2AWrM8Ee2BN7Ya/eo/fsvXnv49YJr5hZz/kfX4B3NKiPQ=</latexit> Review: Gradient Descent in 2D+ gradient = vector of D input: initial θ ∈ R partial derivatives ∂ input: step size α ∈ R +  � ∂θ 0 J r θ J ( θ ) = ∂ ∂θ 1 J while not converged: θ ← θ − α d θ θ � α r θ J ( θ ) d θ J ( θ ) Q: How far to step in that direction? Q: Which direction to step? α · || r θ J ( θ ) || A: A: Straight downhill (steepest descent at current location) Step size parameter picked in advance, unaware of current location Mike Hughes - Tufts COMP 135 - Fall 2020 5

  5. Review: Step size matters Even in one dimension, tough to select step size. 𝑔(𝒚) 𝑔(𝒚) 𝒚 𝒚 𝒚 𝒚 Recommendations - Try multiple values - Might need different sizes at different locations Mike Hughes - Tufts COMP 135 - Fall 2020 6

  6. Review: Neural Net as computational graph 2 directions of propagation Forward: compute loss Backward: compute grad Mike Hughes - Tufts COMP 135 - Fall 2020 7

  7. <latexit sha1_base64="nJnhInwsXMzfqZM21hSK+iPOXxw=">ACGHicbVDLSgMxFM3Ud31VXboJFqEFqTNV0I0giuBKFOwDOnXIpGkbmSGJGMdhvkMN/6KGxeKuHXn35hpu9DqgQsn59xL7j1+yKjStv1l5WZm5+YXFpfyura+uFjc26CiKJSQ0HLJBNHynCqCA1TUjzVASxH1Gv7gPMb90QqGohbHYekzVFP0C7FSBvJK+y7nAovGabQVRH3EnHipHdX8KIEY0/sQbePdBKnpYfsMSzDslco2hV7BPiXOBNSBNce4VPtxPgiBOhMUNKtRw71O0ESU0xI2nejRQJER6gHmkZKhAnqp2MDkvhrlE6sBtIU0LDkfpzIkFcqZj7pMj3VfTXib+57Ui3T1uJ1SEkSYCjz/qRgzqAGYpwQ6VBGsWG4KwpGZXiPtIqxNlnkTgjN98l9Sr1acg0r15rB4ejaJYxFsgx1QAg4AqfgElyDGsDgETyDV/BmPVkv1rv1MW7NWZOZLfAL1uc3L+2ejg=</latexit> Review: Training Neural Nets Training Objective: N X min E ( y n , ˆ y ( x n , w )) w n =1 Gradient Descent Algorithm: w = initialize_weights_at_random_guess(random_state=0) while not converged: total_grad_wrt_w = zeros_like(w) for n in 1, 2, … N: loss[n], grad_wrt_w[n] = forward_and_backward_prop(x[n], y[n], w) total_grad_wrt_w += grad_wrt_w[n] w = w – alpha * total_grad_wrt_w How to pick step size reliably? How to go fast on big datasets? Mike Hughes - Tufts COMP 135 - Fall 2020 8

Recommend

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides
Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner New requirement for the final project: For the first time ever, researchers who submit

619 views • 34 slides
Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade 2018 c University of Washington cse446-staff@cs.washington.edu 1 / 12 Announcements Midterm: Weds, Feb 7th. Policies: You may use a single

333 views • 18 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS 2020 Aaron Mishkin, amishkin@cs.ubc.ca 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of

634 views • 21 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS 2019 Aaron Mishkin 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of the main

1.22k views • 21 slides
Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla December 17, 2011 Peter J. Haas Yannis Sismanis Christina Teflioudi Faraz Makari Outline Matrix Factorization Stochastic Gradient Descent

1.24k views • 75 slides
Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla August 23, 2011 Peter J. Haas Yannis Sismanis Erik Nijkamp Outline Matrix Factorization Stochastic Gradient Descent Distributed SGD with MapReduce

1.05k views • 81 slides
Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background

Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background

Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background In many practical applications, the objective function is a large sum: N f ( x ) = i = 1 f i ( x ) Issues and questions: Evaluating

381 views • 9 slides
CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University of Virginia Overview 1. Gradient Descent 2. Stochastic Gradient Descent 3. SGD with Momentum 4. Adaptive Learning Rates 1 Gradient Descent

767 views • 66 slides
Neural Network Training: Old &amp; New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old &amp; New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old &amp; New Tricks Old: (80s) Stochastic Gradient Descent, Momentum, weight decay New: (last 5-6 years) Dropout ReLUs Batch Normalization Reminder: Overfitting, in images Classification just right

1.61k views • 132 slides
Applied Machine Learning Gradient Descent Methods Siamak Ravanbakhsh COMP 551 (Fall 2020)

Applied Machine Learning Gradient Descent Methods Siamak Ravanbakhsh COMP 551 (Fall 2020)

Applied Machine Learning Gradient Descent Methods Siamak Ravanbakhsh COMP 551 (Fall 2020) Learning objectives Basic idea of gradient descent stochastic gradient descent method of momentum using an adaptive learning rate sub-gradient

568 views • 34 slides
Setting Adaptativity of Stochastic Gradient Descent Aymeric Dieuleveut F. Bach, Non parametric

Setting Adaptativity of Stochastic Gradient Descent Aymeric Dieuleveut F. Bach, Non parametric

Setting Adaptativity of Stochastic Gradient Descent Aymeric Dieuleveut F. Bach, Non parametric stochastic approximation with large step sizes , in the Annals of Statistics Setting : random-design least-squares regression problem in a RKHS

276 views • 3 slides
Large Scale Machine Learning with Stochastic Gradient Descent L eon Bottou leon@bottou.org

Large Scale Machine Learning with Stochastic Gradient Descent L eon Bottou leon@bottou.org

Large Scale Machine Learning with Stochastic Gradient Descent L eon Bottou leon@bottou.org Microsoft (since June) Summary i. Learning with Stochastic Gradient Descent. ii. The Tradeoffs of Large Scale Learning. iii. Asymptotic Analysis.

853 views • 37 slides
Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate Jingfeng Wu , Difan Zou, Vladimir Braverman, Quanquan Gu Johns Hopkins University &amp; UCLA November 2020 Overview

193 views • 16 slides
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects Zhanxing Zhu , Jingfeng Wu , Bing Yu, Lei Wu, Jinwen Ma. Peking University Beijing Institute of Big Data Research

319 views • 15 slides
Variance-based Stochastic Gradient Descent (vSGD): No More Pesky Learning Rates Schaul et al.,

Variance-based Stochastic Gradient Descent (vSGD): No More Pesky Learning Rates Schaul et al.,

Variance-based Stochastic Gradient Descent (vSGD): No More Pesky Learning Rates Schaul et al., ICML13 The idea - Remove need for setting learning rates by updating them optimally from the Hessian values. ADAM: A Method For Stochastic

278 views • 14 slides
Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of

Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of

2020 IEEE International Conference on Acoustics, Speech, and Signal Processing Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers Serge Kas Hanna Email: serge.k.hanna@rutgers.edu Website:

646 views • 16 slides
Stochastic Gradient Descent 10701 Recitations 3 Mu Li Computer Science Department Cargenie

Stochastic Gradient Descent 10701 Recitations 3 Mu Li Computer Science Department Cargenie

Stochastic Gradient Descent 10701 Recitations 3 Mu Li Computer Science Department Cargenie Mellon University February 5, 2013 The problem A typical machine learning problem has a penalty/regularizer + loss form n w F ( w ) = g ( w ) + 1

575 views • 10 slides
Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari

Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari

Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari UCLA VISION LAB 1 The question measures disagreement of predictions with ground truth x = argmin f ( x ) Cat Dog ... x weights aka

496 views • 25 slides
1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

MAP562 Optimal design of structures by Gr egoire Allaire, Thomas Wick Ecole Polytechnique Academic year 2016-2017 Gradient descent and Newtons method for minimization in a PDE / function space context We recapitulate the basic steps to

327 views • 5 slides
On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent Aymeric Dieuleveut

On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent Aymeric Dieuleveut

On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent Aymeric Dieuleveut CMAP, cole Polytechnique, Institut Polytechnique de Paris Joint work with Scott Pesme and Nicolas Flammarion (EPFL) 10/03/2020 Cirm Luminy -

883 views • 53 slides
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated Compression Hanlin Tang, Xiangru Lian , Chen Yu, Tong Zhang, Ji Liu Presenter: Xiangru Lian Compressed SGD (existing algorithms) Worker 1 g (1) g n x

178 views • 7 slides
Interpolation, Growth Conditions, and Stochastic Gradient Descent Aaron Mishkin,

Interpolation, Growth Conditions, and Stochastic Gradient Descent Aaron Mishkin,

Interpolation, Growth Conditions, and Stochastic Gradient Descent Aaron Mishkin, amishkin@cs.ubc.ca 1 45 Training neural networks is dangerous work! 2 45 Chapter 1: Introduction 3 45 Chapter 1: Goal Premise : modern neural networks

703 views • 52 slides
On the Generalization Benefjt of Noise in Stochastic Gradient Descent Samuel L. Smith, Erich

On the Generalization Benefjt of Noise in Stochastic Gradient Descent Samuel L. Smith, Erich

On the Generalization Benefjt of Noise in Stochastic Gradient Descent Samuel L. Smith, Erich Elsen and Soham De ICML 2020 Joint work with Soham De Erich Elsen With thanks to: Esme Sutherland, James Martens, Yee Whye Teh Sander Dieleman,

778 views • 25 slides

More recommend