Neural Network Backpropagation 3-2-16 Recall from Monday... - PowerPoint PPT Presentation

Aug 16, 2023 •451 likes •570 views

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify linearly separable data. Multi-layer networks Can represent any boolean function. We dont want to build them by hand, so we need a way to

Neural Network Backpropagation 3-2-16
Recall from Monday... Perceptrons can only classify linearly separable data.
Multi-layer networks ● Can represent any boolean function. ● We don’t want to build them by hand, so we need a way to train them. ● Algorithm: backpropagation. ○ You’ve already seen this in action in yesterday’s lab.
Backpropagation networks sigmoid ● Backpropagation can be applied to activation functions any directed acyclic neural network. -0.5 ● Activation functions must be 0.2 differentiable. 2.7 0.8 ● Activation functions should be non- 3.0 -0.3 -1.9 linear. -1.2 -1.6 OK not OK 1.5 2.2 0.1 ● Layered networks allow training to be parallelized within each layer.
Sigmoid activation functions ● We want something like a threshold. ○ Neuron is inactive below the threshold; active above it. ● We need something differentiable. ○ Required for gradient descent.
Gradient descent ● Define the squared error at each output node as: ● Update weights to reduce error. ○ Take a step in the direction of steepest descent: w 0 derivative of w 1 learning rate error w.r.t. weight
Computing the error gradient … algebra ensues ...
Gradient descent step for output nodes 2 1.04 2 1 -.97 1.2 -1 1.2
Backpropagation Key idea: at hidden units, use the next-layer change instead of the error function. ● Determine the node’s contribution to its successors. w 0 w 1 ● Update incoming weights using this “error”
Backpropagation algorithm for 1:training runs for example in training_data: run example through network compute error for each output node for each layer (starting from output): for each node in layer: gradient descent update on incoming weights
Exercise: run a backprop step on this network 2 -0.5 0.2 2.7 t = 0.1 0.8 -0.3 3.0 0 -1.2 -1.6 1.5 t = 0.8 0.1 -1

Recommend

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic computation h ( x ) e ( x ) Backpropagation: algorithmic computation of weights M. Magdon-Ismail CSCI 4100/6100 recap: The Neural Network

294 views • 14 slides

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed A brief history of backpropagation The backpropagation algorithm

475 views • 26 slides

Neural Net Backpropagation 3/20/17 Recall: Limitations of Perceptrons vs. AND and OR are

Neural Net Backpropagation 3/20/17 Recall: Limitations of Perceptrons vs. AND and OR are linearly separable. XOR isnt What is the output of the network? ( 0 x < 0 f ( x ) = 1 x 0 1 f ( x ) = 1 + e x ( 0 x < 0 f ( x

714 views • 13 slides

Backpropagation Why backpropagation Neural networks are sequences of parametrized functions

Backpropagation Why backpropagation Neural networks are sequences of parametrized functions x ($; !) linear conv subsample conv subsample filters filters weights Parameters ! Why backpropagation Neural networks are

767 views • 45 slides

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network Perceptron Feed forward Backpropagation Learning rate/momentum What is a neural network? Artificial Human Neurons Each

888 views • 12 slides

Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey

Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, 1 Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others This lecture What is a neural network?

1.1k views • 55 slides

Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural

Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural Network To date: Sharp Straight Sharp Left Ahead Right Neural networks: what are they 30 Output Units Backpropagation: efficient

346 views • 15 slides

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides

Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

1.2k views • 77 slides

Neural Networks + Backpropagation Last Class Softmax Classifier Generalization /

CS4501: Introduction to Computer Vision Neural Networks + Backpropagation Last Class Softmax Classifier Generalization / Overfitting Pytorch Todays Class Global Features The perceptron model Neural Networks

838 views • 34 slides

Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network

Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Neural

462 views • 43 slides

Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network

Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural

712 views • 54 slides

CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1

CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1 / 21 Overview Weve seen that multilayer neural networks are powerful. But how can we actually learn them? Backpropagation is the central

577 views • 21 slides

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth University of Applied Sciences Berlin, Germany 1 Agenda Artificial neuron Activation function Feedforward neural networks Forward

758 views • 39 slides

Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training

Feed-forward Networks Network Training Error Backpropagation Applications Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Neural networks

956 views • 46 slides

Backpropagation and Gradients Agenda Motivation Backprop Tips & Tricks

Backpropagation and Gradients Agenda Motivation Backprop Tips & Tricks Matrix calculus primer Example: 2-layer Neural Network Motivation Recall: Optimization objective is minimize loss Goal: how should we tweak the

428 views • 28 slides

CS4501: Introduction to Computer Vision Deeper Convolutional Neural Network Architectures Last

CS4501: Introduction to Computer Vision Deeper Convolutional Neural Network Architectures Last Class Neural Networks multilayer perceptron model (MLP) Backpropagation Convolutional Neural Networks Todays Class More on

772 views • 41 slides

Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from

Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Backpropagation learning of a network The algorithm 1. Compute a forward pass on the compute graph (DAG) from the input to all the

407 views • 16 slides

Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam

Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam Lopez Credits: Mirella Lapata and Frank Keller 19 January 2018 School of Informatics University of Edinburgh alopez@inf.ed.ac.uk 1 Biological

1.61k views • 96 slides

Backpropagation Learning 15-486/782: Artificial Neural Networks David S. Touretzky Fall 2006 1

Backpropagation Learning 15-486/782: Artificial Neural Networks David S. Touretzky Fall 2006 1 LMS / Widrow-Hoff Rule y w i = y d x i w i x i Works fine for a single layer of trainable weights. What about

695 views • 37 slides

MLPs with Backpropagation CS 472 Backpropagation 1 Multilayer Nets? Linear Systems F(cx) =

MLPs with Backpropagation CS 472 Backpropagation 1 Multilayer Nets? Linear Systems F(cx) = cF(x) F(x+y) = F(x) + F(y) I N M Z Z = (M(NI)) = (MN)I = PI CS 472 Backpropagation 2 Early Attempts Committee Machine Randomly

812 views • 70 slides

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides

Exploiting Randomness in Neural Networks Daniele Di Sarli Mauriana Pesaresi seminars - 2020

Exploiting Randomness in Neural Networks Daniele Di Sarli Mauriana Pesaresi seminars - 2020 Recurrent Neural Network error = (output expected) 2 error W Backpropagation Through Time PREDICTION PATTERNS, INTERACTIONS , 3, 2,

674 views • 25 slides

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation Networks - Slides 2-40 Paul Vicol Shane Baccas George Alexandru Adam Group 2: Priors for Infinite Networks - Slides 41-64

1.37k views • 110 slides