lecture 16 introduction to neural networks feed forward
play

Lecture 16: Introduction to Neural Networks, Feed-forward Networks - PowerPoint PPT Presentation

Jul 13, 2023 •440 likes •1.12k views

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

  1. Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu

  2. Recap Previous Lecture 2 C. Long Lecture 16 April 22, 2018

  3. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 3 C. Long Lecture 16 April 22, 2018

  4. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 4 C. Long Lecture 16 April 22, 2018

  5. Neural networks Neural networks are made up of many artificial neurons . • Each input into the neuron has its own weight • associated with it illustrated by the red circle . A weight is simply a floating point number and it ' s these • we adjust when we eventually come to train the network . 5 C. Long Lecture 16 April 22, 2018

  6. Neural networks A neuron can have any number of inputs from one to n, • where n is the total number of inputs. The inputs may be represented therefore as x 1 , x 2 , x 3 … • x n . And the corresponding weights for the inputs as w 1 , w 2 , • w 3 … w n . Output a = x 1 w 1 +x 2 w 2 +x 3 w 3 ... +x n w n • 6 C. Long Lecture 16 April 22, 2018

  7. McCulloch–Pitts “unit” Output is a “squashed” linear function of the inputs : • A gross oversimplification of real neurons , but its • purpose is to develop understanding of what networks of simple units can do 7 C. Long Lecture 16 April 22, 2018

  8. Activation functions 8 C. Long Lecture 16 April 22, 2018

  9. Neural Networks by an Example let ' s design a neural network that will detect the number '4'. • Given a panel made up of a grid of lights which can be either on or • off , we want our neural net to let us know whenever it thinks it sees the character '4'. The panel is eight cells square and looks like this : • the neural net will have 64 inputs , each one representing a • particular cell in the panel and a hidden layer consisting of a number of neurons ( more on this later ) all feeding their output into just one neuron in the output layer 9 9 C. Long Lecture 16 April 22, 2018

  10. Neural Networks by an Example Initialize the neural net with random weights • Feed it a series of inputs which represent , in this • example , the different panel configurations For each configuration we check to see what its output • is and adjust the weights accordingly so that whenever it sees something looking like a number 4 it outputs a 1 and for everything else it outputs a zero . http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html 10 C. Long Lecture 16 April 22, 2018

  11. A Cheat Sheet of Neural Network Architectures 11 C. Long Lecture 16 April 22, 2018

  12. A Cheat Sheet of Neural Network Architectures 12 C. Long Lecture 16 April 22, 2018

  13. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 13 C. Long Lecture 16 April 22, 2018

  14. Network structures Feed - forward networks : • – single - layer perceptrons – multi - layer perceptrons Feed - forward networks implement functions , have no • internal state 14 C. Long Lecture 16 April 22, 2018

  15. Feed-forward example Feed - forward network = a parameterized family of • nonlinear functions : Adjusting weights changes the function : do learning • this way ! 15 C. Long Lecture 16 April 22, 2018

  16. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 16 C. Long Lecture 16 April 22, 2018

  17. Single-layer perceptrons Output units all operate separately—no shared weights • Adjusting weights moves the location , orientation , and • steepness of cliff 17 C. Long Lecture 16 April 22, 2018

  18. Expressiveness of perceptrons Consider a perceptron with g = step function • ( Rosenblatt , 1957, 1960) Can represent AND , OR , NOT , majority , etc ., but not • XOR Represents a linear separator in input space : • Minsky & Papert (1969) pricked the neural network balloon 18 C. Long Lecture 16 April 22, 2018

  19. Perceptron learning 19 C. Long Lecture 16 April 22, 2018

  20. Perceptron learning contd. Perceptron learning rule converges to a consistent • function for any linearly separable data set Perceptron learns majority function easily, DTL is hopeless DTL learns restaurant function easily, perceptron cannot represent it. 20 C. Long Lecture 16 April 22, 2018

  21. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 21 C. Long Lecture 16 April 22, 2018

  22. Multilayer perceptrons Layers are usually fully connected , and numbers of • hidden units typically chosen by hand 22 C. Long Lecture 16 April 22, 2018

  23. Multi-Layer Perceptron We will introduce the MLP and the backpropagation • algorithm which is used to train it MLP used to describe any general feedforward (no recurrent • connections) network However, we will concentrate on nets with units arranged in • layers x 1 x n 23 C. Long Lecture 16 April 22, 2018

  24. Multi-Layer Perceptron x 1 x n Different books refer to the above as either 4 layer (no. of • layers of neurons) or 3 layer (no. of layers of adaptive weights). We will follow the latter convention. What do the extra layers gain you? Start with looking at • what a single layer can’t do. 24 C. Long Lecture 16 April 22, 2018

  25. How do we actually use an artificial neuron? Feedforward network: The neurons in each layer feed • their output forward to the next layer until we get the final output from the neural network. There can be any number of hidden layers within a • feedforward network. The number of neurons can be completely arbitrary. • 25 C. Long Lecture 16 April 22, 2018

  26. Perceptron Learning Theorem Recap : A perceptron ( threshold unit ) can learn • anything that it can represent ( i . e . anything separable with a hyperplane ) Logical OR Function x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 1 26 C. Long Lecture 16 April 22, 2018

  27. The Exclusive OR problem A Perceptron cannot represent Exclusive OR since it is not linearly separable . Logical XOR Function x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 0 27 27 C. Long Lecture 16 April 22, 2018

  28. Piecewise linear classification using an MLP Minsky & Papert (1969) offered solution to XOR problem • by combining perceptron unit responses using a second layer of Units. Piecewise linear classification using an MLP with threshold (perceptron) units 28 C. Long Lecture 16 April 22, 2018

  29. Piecewise linear classification using an MLP Three-layer networks 29 C. Long Lecture 16 April 22, 2018

  30. Properties of architecture No connections within a layer • No direct connections between input and output layers • Fully connected between layers • Often more than 3 layers • Number of output units need not equal number of input units • Number of hidden units per layer can be more or less than input or • output units Each unit is a perceptron m = + å y f ( w x b ) i i j j i = j 1 30 C. Long Lecture 16 April 22, 2018

  31. What do each of the layers do? 1st layer draws linear 2nd layer combines the 3rd layer can generate arbitrarily boundaries boundaries complex boundaries 31 C. Long Lecture 16 April 22, 2018

  32. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 32 C. Long Lecture 16 April 22, 2018

  33. Backpropagation learning algorithm BP Solution to credit assignment problem in MLP. Rumelhart, • Hinton and Williams (1986) ( though actually invented earlier in a PhD thesis relating to economics) BP has two phases : • Forward pass phase : computes ‘functional signal’, feed forward propagation of input pattern signals through network. Backward pass phase : computes ‘error signal’, propagates the error backwards through network starting at output units (where the error is the difference between actual and desired output values) 33 C. Long Lecture 16 April 22, 2018

  34. Conceptually: Forward Activity - Backward Error Link between hidden node j Output node i and output node i: Wji Hidden node j Link between input node k Input node k and hidden node j: Wkj 34 C. Long Lecture 16 April 22, 2018

  35. Back-propagation derivation The squared error on a single example is defined as • where the sum is over the nodes in the output layer. 35 C. Long Lecture 16 April 22, 2018

  36. Back-propagation derivation contd. 36 C. Long Lecture 16 April 22, 2018

  37. Back-propagation Learning Output layer: same as the single-layer perceptron • where Hidden layer: back-propagation the error from the output • layer. Upadate rules for weights in the hidden layers. • (Most neuroscientists deny that back-propagation occurs in the brain) 37 C. Long Lecture 16 April 22, 2018

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
2nd Session Machine learning: feed-forward neural networks and self-organizing maps 1

2nd Session Machine learning: feed-forward neural networks and self-organizing maps 1

2nd Session Machine learning: feed-forward neural networks and self-organizing maps 1 Recommended reading J. Zupan, J. Gasteiger, Neural Networks in Chemistry and Drug Design: An Introduction, Wiley-VCH, Weinheim, 1999.

959 views • 62 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Todays Lecture Feed-forward neural networks as classifiers simple architecture in which

724 views • 37 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets -

Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets -

Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets - Self-Organizing Maps - Feed-Forward Neural Networks - Backpropagation Overview 1 2 Introduction Idea: Mimic principle of biological neural networks

468 views • 34 slides
CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Variable length data Traditional feed forward neural networks can only handle fixed

609 views • 18 slides
Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network

Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network

Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Neural

462 views • 43 slides
Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network

Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network

Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural

712 views • 54 slides
Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training

Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training

Feed-forward Networks Network Training Error Backpropagation Applications Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Neural networks

956 views • 46 slides
Neural Networks: Introduction LING572 Advanced Statistical Methods for NLP February 25 2020 1

Neural Networks: Introduction LING572 Advanced Statistical Methods for NLP February 25 2020 1

Neural Networks: Introduction LING572 Advanced Statistical Methods for NLP February 25 2020 1 Unit Overview Introduction: History; Main Ideas / Basic Computation; Landscape Computation in feed-forward networks + Beginning of Learning

516 views • 41 slides
1 Kinds of Networks Feed-forward Single layer Multi-layer Recurrent Kinds of

1 Kinds of Networks Feed-forward Single layer Multi-layer Recurrent Kinds of

CSE 592 Applications of Artificial Intelligence Neural Networks & Data Mining Henry Kautz Winter 2003 1 Kinds of Networks Feed-forward Single layer Multi-layer Recurrent Kinds of Networks Kinds of Networks

308 views • 18 slides
Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap from last time Feed-Forward Neural Network: Multilayer Perceptron + 0 ) = (

1.04k views • 93 slides
MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

Run 3493 Event 41075, Oct. 23rd, 2015 CONVOLUTIONAL NEURAL NETWORKS IN MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks (CNNs) are a type of deep, feed-forward neural networks that have been

831 views • 43 slides
L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And

L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And

L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And their limitations: http://www.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node19.html What if we could use multiple classifiers?

579 views • 19 slides
The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University

The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University

The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University November 15, 2019 Overview Review of Perceptron and Feed Forward Networks Recurrent Neural Networks Neural Turing Machines

1.27k views • 57 slides
Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network Perceptron Feed forward Backpropagation Learning rate/momentum What is a neural network? Artificial Human Neurons Each

889 views • 12 slides
Lecture 7 Introduction to Neural Networks Julia Hockenmaier juliahmr@illinois.edu 3324

Lecture 7 Introduction to Neural Networks Julia Hockenmaier juliahmr@illinois.edu 3324

CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Lecture 7 Introduction to Neural Networks Julia Hockenmaier juliahmr@illinois.edu 3324 Siebel Center Lecture 7: Introduction to Neural Networks : 1 t r a

669 views • 28 slides
6. Feed-forward mapping networks Fundamentals of Computational Neuroscience, T. P. Trappenberg,

6. Feed-forward mapping networks Fundamentals of Computational Neuroscience, T. P. Trappenberg,

6. Feed-forward mapping networks Fundamentals of Computational Neuroscience, T. P. Trappenberg, 2002. Lecture Notes on Brain and Computation Byoung-Tak Zhang Biointelligence Laboratory School of Computer Science and Engineering Graduate

645 views • 28 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
CS4501: Introduction to Computer Vision Neural Networks (NNs) Artificial Neural Networks (ANNs)

CS4501: Introduction to Computer Vision Neural Networks (NNs) Artificial Neural Networks (ANNs)

CS4501: Introduction to Computer Vision Neural Networks (NNs) Artificial Neural Networks (ANNs) Multi-layer Perceptrons (MLPs Previous Neural Networks The Perceptron Model The Multi-layer Perceptron (MLP) Forward-pass in an MLP

1.54k views • 40 slides

More recommend