lecture 4 backpropagation and neural networks part 1
play

Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li - PowerPoint PPT Presentation

Nov 04, 2023 •398 likes •1.26k views

Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 1 Administrative A1 is due

  1. Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 1

  2. Administrative A1 is due Jan 20 (Wednesday). ~150 hours left Warning: Jan 18 (Monday) is Holiday (no class/office hours) Also note: Lectures are non-exhaustive. Read course notes for completeness. I’ll hold make up office hours on Wed Jan20, 5pm @ Gates 259 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 2

  3. Where we are... scores function SVM loss data loss + regularization want Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 3

  4. Optimization (image credits to Alec Radford) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 4

  5. Gradient Descent Numerical gradient : slow :(, approximate :(, easy to write :) Analytic gradient : fast :), exact :), error-prone :( In practice: Derive analytic gradient, check your implementation with numerical gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 5

  6. Computational Graph x s (scores) * hinge L + loss W R Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 6

  7. Convolutional Network (AlexNet) input image weights loss Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 7

  8. Neural Turing Machine input tape loss Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 8

  9. Neural Turing Machine Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 9

  10. e.g. x = -2, y = 5, z = -4 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 10

  11. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 11

  12. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 12

  13. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 13

  14. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 14

  15. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 15

  16. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 16

  17. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 17

  18. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 18

  19. e.g. x = -2, y = 5, z = -4 Chain rule: Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 19

  20. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 20

  21. e.g. x = -2, y = 5, z = -4 Chain rule: Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 21

  22. activations f Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 22

  23. activations “local gradient” f Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 23

  24. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 24

  25. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 25

  26. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 26

  27. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 27

  28. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 28

  29. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 29

  30. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 30

  31. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 31

  32. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 32

  33. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 33

  34. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 34

  35. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 35

  36. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 36

  37. Another example: (-1) * (-0.20) = 0.20 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 37

  38. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 38

  39. Another example: [local gradient] x [its gradient] [1] x [0.2] = 0.2 [1] x [0.2] = 0.2 (both inputs!) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 39

  40. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 40

  41. Another example: [local gradient] x [its gradient] x0: [2] x [0.2] = 0.4 w0: [-1] x [0.2] = -0.2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 41

  42. sigmoid function sigmoid gate Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 42

  43. sigmoid function sigmoid gate (0.73) * (1 - 0.73) = 0.2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 43

Recommend

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed A brief history of backpropagation The backpropagation algorithm

475 views • 26 slides
Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic computation h ( x ) e ( x ) Backpropagation: algorithmic computation of weights M. Magdon-Ismail CSCI 4100/6100 recap: The Neural Network

294 views • 14 slides
Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam

Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam

Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam Lopez Credits: Mirella Lapata and Frank Keller 19 January 2018 School of Informatics University of Edinburgh alopez@inf.ed.ac.uk 1 Biological

1.61k views • 96 slides
CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1

CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1

CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1 / 21 Overview Weve seen that multilayer neural networks are powerful. But how can we actually learn them? Backpropagation is the central

577 views • 21 slides
Backpropagation Why backpropagation Neural networks are sequences of parametrized functions

Backpropagation Why backpropagation Neural networks are sequences of parametrized functions

Backpropagation Why backpropagation Neural networks are sequences of parametrized functions x ($; !) linear conv subsample conv subsample filters filters weights Parameters ! Why backpropagation Neural networks are

767 views • 45 slides
Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Backpropagation Now we consider ERM problem of minimizing the following empirical risk function over : n R (

421 views • 3 slides
Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey

Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey

Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, 1 Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others This lecture What is a neural network?

1.1k views • 55 slides
Neural Networks + Backpropagation Last Class Softmax Classifier Generalization /

Neural Networks + Backpropagation Last Class Softmax Classifier Generalization /

CS4501: Introduction to Computer Vision Neural Networks + Backpropagation Last Class Softmax Classifier Generalization / Overfitting Pytorch Todays Class Global Features The perceptron model Neural Networks

838 views • 34 slides
Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob

Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob

Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob CPSC 533 Winter 2001 Learning by Gradient Descent Definition of the Learning Problem Let us start with the simple case of linear cells, which

558 views • 22 slides
An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth University of Applied Sciences Berlin, Germany 1 Agenda Artificial neuron Activation function Feedforward neural networks Forward

758 views • 39 slides
CS7015 (Deep Learning): Lecture 4 Feedforward Neural Networks, Backpropagation Mitesh M. Khapra

CS7015 (Deep Learning): Lecture 4 Feedforward Neural Networks, Backpropagation Mitesh M. Khapra

CS7015 (Deep Learning): Lecture 4 Feedforward Neural Networks, Backpropagation Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/9 Mitesh M. Khapra CS7015 (Deep Learning): Lecture 4

1.85k views • 59 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

1.2k views • 77 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
CS489/698 Lecture 9: Feb 1, 2017 Multi-layer Neural Networks, Error Backpropagation [D] Chapt.

CS489/698 Lecture 9: Feb 1, 2017 Multi-layer Neural Networks, Error Backpropagation [D] Chapt.

CS489/698 Lecture 9: Feb 1, 2017 Multi-layer Neural Networks, Error Backpropagation [D] Chapt. 10, [HTF] Chapt. 11, [B] Sec. 5.2, 5.3, [M] Sec. 16.5, [RN] Sec. 18.7 CS489/698 (c) 2017 P. Poupart 1 Quick Recap: Linear Models Linear

604 views • 18 slides
Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Neural Natural Language Processing Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part 1 : Language modeling. Part 2 : Recurrent neural networks. Part 3 : Long-Short Term Memory (LSTM).

1.07k views • 78 slides
Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation Networks - Slides 2-40 Paul Vicol Shane Baccas George Alexandru Adam Group 2: Priors for Infinite Networks - Slides 41-64

1.37k views • 110 slides
Backpropagation Learning 15-486/782: Artificial Neural Networks David S. Touretzky Fall 2006 1

Backpropagation Learning 15-486/782: Artificial Neural Networks David S. Touretzky Fall 2006 1

Backpropagation Learning 15-486/782: Artificial Neural Networks David S. Touretzky Fall 2006 1 LMS / Widrow-Hoff Rule y w i = y d x i w i x i Works fine for a single layer of trainable weights. What about

695 views • 37 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron

Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron

CS4501: Introduction to Computer Vision Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron model Neural Networks multilayer perceptron model (MLP) Backpropagation Todays Class

789 views • 34 slides
343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT Austin Today Neural networks Supervised learning in visual recognition What does recognition involve? Verification: is that a lamp?

774 views • 49 slides
Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify linearly separable data. Multi-layer networks Can represent any boolean function. We dont want to build them by hand, so we need a way to

570 views • 11 slides
Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets -

Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets -

Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets - Self-Organizing Maps - Feed-Forward Neural Networks - Backpropagation Overview 1 2 Introduction Idea: Mimic principle of biological neural networks

468 views • 34 slides

More recommend