neural networks
play

Neural Networks Luke Zettlemoyer (Slides adapted from Danqi Chen, - PowerPoint PPT Presentation

Nov 22, 2022 •207 likes •632 views

CSEP 517 Natural Language Processing Neural Networks Luke Zettlemoyer (Slides adapted from Danqi Chen, Chris Manning, Dan Jurafsky) Neural networks for NLP Feed-forward NNs Recurrent NNs Transformer Convolutional NNs Always coupled with

  1. CSEP 517 Natural Language Processing Neural Networks Luke Zettlemoyer (Slides adapted from Danqi Chen, Chris Manning, Dan Jurafsky)

  2. Neural networks for NLP Feed-forward NNs Recurrent NNs Transformer Convolutional NNs Always coupled with word embeddings…

  3. This Lecture • Feedforward Neural Networks • Applications • Neural Bag-of-Words Models • Feedforward Neural Language Models • The training algorithm: Back-propagation

  4. Neural Networks: History

  5. NN “dark ages” • Neural network algorithms date from the 80s • ConvNets: applied to MNIST by LeCun in 1998 • Long Short-term Memory Networks (LSTMs): Hochreiter and Schmidhuber 1997 • Henderson 2003: neural shift-reduce parser, not SOTA Credits: Greg Durrett

  6. 2008-2013: A glimmer of light • Collobert and Weston 2011: “ NLP (almost) from Scratch ” • Feedforward NNs can replace “feature engineering” • 2008 version was marred by bad experiments, claimed SOTA but wasn’t, 2011 version tied SOTA • Krizhevskey et al, 2012: AlexNet for ImageNet Classification • Socher 2011-2014: tree-structured RNNs working okay Credits: Greg Durrett

  7. 2014: Stuff starts working • Kim (2014) + Kalchbrenner et al, 2014: sentence classification • ConvNets work for NLP! • Sutskever et al, 2014: sequence-to-sequence for neural MT • LSTMs work for NLP! • Chen and Manning 2014: dependency parsing • Even feedforward networks work well for NLP! • 2015: explosion of neural networks for everything under the sun Credits: Greg Durrett

  8. Why didn’t they work before? • Datasets too small : for MT, not really better until you have 1M+ parallel sentences (and really need a lot more) • Optimization not well understood : good initialization, per- feature scaling + momentum (Adagrad/Adam) work best out-of- the-box • Regularization: dropout is pretty helpful • Computers not big enough: can’t run for enough iterations • Inputs: need word embeddings to represent continuous semantics Credits: Greg Durrett

  9. The “Promise” • Most NLP works in the past focused on human-designed representations and input features • Representation learning attempts to automatically learn good features and representations • Deep learning attempts to learn multiple levels of representation on increasing complexity/abstraction

  10. Feed-forward Neural Networks

  11. Feed-forward NNs • Input: x 1 , …, x d • Output: y ∈ {0,1}

  12. Neural computation Computation units: neurons

  13. An artificial neuron • A neuron is a computational unit that has scalar inputs and an output • Each input has an associated weight. • The neuron multiples each input by its weight, sums them, applied a nonlinear function to the result, and passes it to its output.

  14. Neural networks • The neurons are connected to each other, forming a network • The output of a neuron may feed into the inputs of other neurons

  15. <latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit> <latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit> <latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit> <latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit> <latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit> <latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit> <latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit> <latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit> A neuron can be a binary logistic regression unit 1 f ( z ) = 1 + e − z h w ,b ( x ) = f ( w | x + b )

  16. A neural network = many layers of classifiers all learned at once, some providing features for others • If we feed a vector of inputs through a bunch of logistic regression functions, then we get a vector of outputs… • which we can feed into another logistic regression function

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., &quot;Neural networks and physical systems with emergent collective computational abilities,&quot; Proceedings of the National Academy

367 views • 19 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides
Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC www.kit.edu KIT The Research University in the Helmholtz Association Neural Network Concept (very) remotely brain inspired computational system

479 views • 25 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand intricacies of protein, cell, and network function within the brain, neuroscientists would agree on the following simplistic description of how the

157 views • 11 slides
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons

Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons

Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of &gt; 20 types, 10 14 synapses, 1ms10ms cycle

804 views • 47 slides
Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A

Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A

Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Why Model Neural Networks? The nervous system is a

1.22k views • 23 slides
Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif f erent models of neural net w orks combined int o a single syst em. Each single net w ork is made int o a module t hat can be f reely int

289 views • 25 slides
Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0 L0 M*V Processor N N N N N N Project Brainwave Pretrained DNN Model Scalable DNN Hardware Neural Processing Unit Microservice Extract

525 views • 30 slides
Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven

Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven

Neural Networks and Handwriting Recognition Steven Sloss Math 164 Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven Sloss Structure Training Neural Networks Math 164 Motivation Problem

888 views • 41 slides

More recommend