lecture 6 training neural networks part 2
play

Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li & - PowerPoint PPT Presentation

Dec 13, 2023 •408 likes •1.28k views

Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 1 Administrative A2 is out. Its

  1. Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 1

  2. Administrative A2 is out. It’s meaty. It’s due Feb 5 (next Friday) You’ll implement: Neural Nets (with Layer Forward/Backward API) Batch Norm Dropout ConvNets Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 2

  3. Mini-batch SGD Loop: 1. Sample a batch of data 2. Forward prop it through the graph, get loss 3. Backprop to calculate the gradients 4. Update the parameters using the gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 3

  4. Leaky ReLU Activation Functions max(0.1x, x) Sigmoid Maxout tanh tanh(x) ELU ReLU max(0,x) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 4

  5. Data Preprocessing Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 5

  6. “Xavier initialization” [Glorot et al., 2010] Reasonable initialization. (Mathematical derivation assumes linear activations) Weight Initialization Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 6

  7. [Ioffe and Szegedy, 2015] Batch Normalization - Improves gradient flow Normalize: through the network - Allows higher learning rates - Reduces the strong dependence on initialization And then allow the network to squash - Acts as a form of the range if it wants to: regularization in a funny way, and slightly reduces the need for dropout, maybe Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 7

  8. Babysitting the Cross-validation learning process Loss barely changing: Learning rate is probably too low Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 8

  9. TODO - Parameter update schemes - Learning rate schedules - Dropout - Gradient checking - Model ensembles Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 9

  10. Parameter Updates Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 10

  11. Training a neural network, main loop: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 11

  12. Training a neural network, main loop: simple gradient descent update now: complicate. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 12

  13. Image credits: Alec Radford Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 13

  14. Suppose loss function is steep vertically but shallow horizontally: Q: What is the trajectory along which we converge towards the minimum with SGD? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 14

  15. Suppose loss function is steep vertically but shallow horizontally: Q: What is the trajectory along which we converge towards the minimum with SGD? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 15

  16. Suppose loss function is steep vertically but shallow horizontally: Q: What is the trajectory along which we converge towards the minimum with SGD? very slow progress along flat direction, jitter along steep one Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 16

  17. Momentum update - Physical interpretation as ball rolling down the loss function + friction (mu coefficient). - mu = usually ~0.5, 0.9, or 0.99 (Sometimes annealed over time, e.g. from 0.5 -> 0.99) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 17

  18. Momentum update - Allows a velocity to “build up” along shallow directions - Velocity becomes damped in steep direction due to quickly changing sign Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 18

  19. SGD vs Momentum notice momentum overshooting the target, but overall getting to the minimum much faster. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 19

  20. Nesterov Momentum update Ordinary momentum update: momentum step actual step gradient step Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 20

  21. Nesterov Momentum update Nesterov momentum update Momentum update “lookahead” gradient step (bit different than momentum momentum original) step step actual step actual step gradient step Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 21

  22. Nesterov Momentum update Nesterov momentum update Momentum update “lookahead” gradient step (bit different than momentum momentum original) step step actual step actual step gradient Nesterov: the only difference... step Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 22

  23. Nesterov Momentum update Slightly inconvenient… usually we have : Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 23

  24. Nesterov Momentum update Slightly inconvenient… usually we have : Variable transform and rearranging saves the day: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 24

  25. Nesterov Momentum update Slightly inconvenient… usually we have : Variable transform and rearranging saves the day: Replace all thetas with phis, rearrange and obtain: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 25

  26. nag = Nesterov Accelerated Gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 26

  27. [Duchi et al., 2011] AdaGrad update Added element-wise scaling of the gradient based on the historical sum of squares in each dimension Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 27

  28. AdaGrad update Q: What happens with AdaGrad? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 28

  29. AdaGrad update Q2: What happens to the step size over long time? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 29

  30. [Tieleman and Hinton, 2012] RMSProp update Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 30

  31. Introduced in a slide in Geoff Hinton’s Coursera class, lecture 6 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 31

  32. Introduced in a slide in Geoff Hinton’s Coursera class, lecture 6 Cited by several papers as: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 32

  33. adagrad rmsprop Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 33

Recommend

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019 Reminder: A3 Due Monday, October 14 (1 week from today!) Remember to run the validation script! Justin Johnson Lecture 10 - 2 October 7, 2019

1.45k views • 101 slides
Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1

Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1

Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1 Reminder: A3 Due Friday, October 9 Justin Johnson October 5, 2020 Lecture 10 - 2 Midterm Exam We are still working out details! Will share more

1.31k views • 102 slides
Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

1.79k views • 165 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Neural Natural Language Processing Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part 1 : Language modeling. Part 2 : Recurrent neural networks. Part 3 : Long-Short Term Memory (LSTM).

1.07k views • 78 slides
TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides
Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin

Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin

Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 1 Administrative A1 is due today

1.36k views • 102 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT Austin Today Neural networks Supervised learning in visual recognition What does recognition involve? Verification: is that a lamp?

774 views • 49 slides
Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active learning - How transferable are features in deep neural When you are not learning networks? Surrogat losses

1k views • 69 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have

Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have

CSCI 5525 Machine Learning Fall 2019 Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have just learned about kernel functions that allow us to implicitly lift the raw feature vector x to expanded

371 views • 5 slides
Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides
Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Backpropagation Now we consider ERM problem of minimizing the following empirical risk function over : n R (

421 views • 3 slides
Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear models for classification Predictions are made as a sequence of simple operations matrix-vector operations non-linear activation functions

417 views • 14 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Neural Networks Learning the network: Part 1 11-785, Fall 2020 Lecture 3 1 Topics for the day

Neural Networks Learning the network: Part 1 11-785, Fall 2020 Lecture 3 1 Topics for the day

Neural Networks Learning the network: Part 1 11-785, Fall 2020 Lecture 3 1 Topics for the day The problem of learning The perceptron rule for perceptrons And its inapplicability to multi-layer perceptrons Greedy solutions for

1.96k views • 160 slides
Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark

Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark

Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark Glickman and Chris Tanner 1 Outline Review from last lecture 1. Training CNNs 2. BackProp of MaxPooling layer 3. Layers Receptive Field 4.

2.22k views • 89 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides
DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi , Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, Martin Vechev github.com/eth-sri/dl2 differencing neural networks finding inputs that deactivates [Pei

992 views • 13 slides
Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu Universal Approximators Neural networks can approximate [1] function. any Capacity Layers hidden layer size

1.19k views • 17 slides

More recommend