language modeling with gated convolutional networks
play

LANGUAGE MODELING WITH GATED CONVOLUTIONAL NETWORKS YANN N. - PowerPoint PPT Presentation

Jan 06, 2023 •350 likes •561 views

LANGUAGE MODELING WITH GATED CONVOLUTIONAL NETWORKS YANN N. DAUPHIN, ANGELA FAN, MICHAEL AULI AND DAVID GRANGIER FACEBOOK AI RESEARCH CS 546 Paper Presentation Jinfeng Xiao 2/22/2018 Intro: Language Models Full$model: / * + , , , + /

  1. LANGUAGE MODELING WITH GATED CONVOLUTIONAL NETWORKS YANN N. DAUPHIN, ANGELA FAN, MICHAEL AULI AND DAVID GRANGIER FACEBOOK AI RESEARCH CS 546 Paper Presentation Jinfeng Xiao 2/22/2018

  2. Intro: Language Models ■ Full$model: / * + , , … , + / = * + , 1 * + 2 |+ , , … , + 264 234 ■ n-gram model: * + 2 = * + 2 |+ 26784 , … , + 264 ■ Hard to represent long-range dependencies, due to data sparsity

  3. “Gate” Intro: LSTM http://colah.github.io/posts/2015-08-Understanding-LSTMs/

  4. Intro: LSTM ■ State-of-the-art neural network approach for language modeling ■ + Can theoretically model arbitrarily long dependencies ■ -- Not parallelizable; O(N) operations http://colah.github.io/posts/2015-08-Understanding-LSTMs

  5. Intro: CNN k ■ Predict the current word y with previous words x (i.e. context) ■ Model long-term dependencies with O(N/k) operations

  6. This Paper: GCNN ■ Gated Convolutional Neural Networks ■ Each CNN layer is followed by a gating layer ■ Allows parallelization over sequential tokens ■ Reduces the latency to score a sentence by an order of magnitude ■ Competitive performance on WikiText-103 and Google Billion Words benchmarks

  7. Architecture ■ Word Embedding + ■ CNN + ■ Gating

  8. Architecture ■ Word E Embedding + ■ CNN + ■ Gating

  9. Architecture ■ Word Embedding + ■ CNN CNN + ■ Gating *: Convolution operation

  10. Architecture ■ Word Embedding + ■ CNN CNN + ■ Gating learned parameters

  11. Example: Convolution ■ “Average” over a small patch around an element http://colah.github.io/posts/2015-08-Understanding-LSTMs

  12. Architecture ■ Word Embedding + ■ CNN + ■ Ga Gating

  13. Two Gating Mechanisms ■ Gated linear units (GLU) ℎ " # = # ∗ & + ( ⊗σ # ∗ + + , ■ Gated tanh units (GTU) ℎ - # = tanh # ∗ & + ( ⊗σ # ∗ + + ,

  14. Evaluation Metric: Perplexity ■ The perplexity of a discrete probability distribution p is " ( # ∑ %&' )*+,- . % |…,. %2' ! ■ It measures how well our model matches the held out test data set. ■ The smaller, the better. https://en.wikipedia.org/wiki/Perplexity

  15. Benchmark: Google Billion Word Average Sequence Length = 20 Words ReLU % = %⊗ % > 0

  16. GCNN Is Faster On Google Billion Words

  17. Benchmark: WikiText-103 Average Sequence Length = 4,000 Words

  18. Short Context Size Suffices Google Billion Word Wiki-103 Avg. Text Length = 20 Avg. Text Length = 4,000

  19. Summary ■ GCNN: CNN + Gating ■ Perplexity is comparable with the state-of-the-art LSTM ■ GCNN converges faster and allows parallelization over sequential tokens ■ The simpler linear gating (GLU) works better than LSTM-like tanh gating (GTU)

Recommend

Outline Gated Feedback Recurrent Neural Networks. arXiv1502. Introduction: RNN & Gated RNN

Outline Gated Feedback Recurrent Neural Networks. arXiv1502. Introduction: RNN & Gated RNN

Outline GF-RNN ReNet Outline Gated Feedback Recurrent Neural Networks. arXiv1502. Introduction: RNN & Gated RNN Gated Feedback Recurrent Neural Networks (GF-RNN) Experiments: Character-level Language Modeling & Python Program

631 views • 26 slides
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides
C Crowd Counting and Behavior d C ti d B h i Modeling with Modeling with Convolutional Neural

C Crowd Counting and Behavior d C ti d B h i Modeling with Modeling with Convolutional Neural

C Crowd Counting and Behavior d C ti d B h i Modeling with Modeling with Convolutional Neural Networks Hongsheng Li Hongsheng Li 1 Dept. of Electronic Enigineering, 2 Multimedia Laboratory The Chinese University of Hong

666 views • 52 slides
Outline Introduction Convolutional neural networks (CNN) The usage of intermediate

Outline Introduction Convolutional neural networks (CNN) The usage of intermediate

23rd International Conference on MultiMedia Modeling (MMM 2017) On the Exploration of Convolutional Fusion Networks for Visual Recognition Yu Liu, Yanming Guo, and Michael S. Lew Leiden Institute of Advanced Computer Science, Leiden University

1.06k views • 51 slides
Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease Predictions Narges

Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease Predictions Narges

Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease Predictions Narges Razavian New York University Langone Medical Center GTC 2017 In collaboration with: David Sontag PhD , Saul Blecker MD , Ann-Marie Schmidt MD ,

480 views • 28 slides
Extracting and Modeling Relations with Graph Convolutional Networks Ivan Titov with Diego

Extracting and Modeling Relations with Graph Convolutional Networks Ivan Titov with Diego

Extracting and Modeling Relations with Graph Convolutional Networks Ivan Titov with Diego Marcheggiani, Michael Schlichtkrull, Thomas Kipf, Max Welling, Rianne van den Berg and Peter Bloem 1 Inferring missing facts in knowledge bases: link

827 views • 66 slides
Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides
IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks (2:2)

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks (2:2)

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks (2:2) Erik Velldal & Lilja vrelid University of Oslo 3 March 2020 Agenda Brief recap from last week on CNNs. Extensions of the basic CNN

592 views • 34 slides
Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN) A.k.a. CNN or ConvNet Adit Deshpande, A Beginner's Guide To Understanding Convolutional Neural Networks. Digital Images Input array: an images

1.42k views • 72 slides
15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter (this lecture) and Ariel Procaccia Carnegie Mellon University Spring 2017 1 Outline Convolutional neural networks Applications of convolutional

1.32k views • 32 slides
Convolutional Neural Networks for Language CS 6956: Deep Learning for NLP Features from text

Convolutional Neural Networks for Language CS 6956: Deep Learning for NLP Features from text

Convolutional Neural Networks for Language CS 6956: Deep Learning for NLP Features from text Example: Sentiment classification The goal: Is the sentiment of a sentence positive, negative or neutral? The film is fun and is host to some truly

1.37k views • 88 slides
Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How do we evaluate language models? Traditional language models Feedforward neural networks for language modeling Recurrent neural networks for

1.58k views • 58 slides
IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik Velldal University of Oslo 25 February 2020 So far: MLPs + embeddings as inputs Embeddings have benefits over discrete feature vectors; makes

902 views • 35 slides
IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik Velldal University of Oslo 25 February 2020 So far: MLPs + embeddings as inputs Embeddings have benefits over discrete feature vectors; makes

806 views • 58 slides
Gated Path Planning Networks Lisa Lee Machine Learning Department Carnegie Mellon University

Gated Path Planning Networks Lisa Lee Machine Learning Department Carnegie Mellon University

Gated Path Planning Networks Lisa Lee Machine Learning Department Carnegie Mellon University Joint work with Emilio Parisotto, Devendra Chaplot, Eric Xing, & Ruslan Salakhutdinov ICML 2018 Path Planning Gated Path Planning Networks

280 views • 26 slides
Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides
and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

Band-limited Training and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast Training of Convolutional Networks through FFTs Fast Convolutional Nets With fbfft: A GPU Performance Evaluation Data: x

648 views • 46 slides
Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks

Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks

Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks Navid Rekab-Saz navid.rekabsaz@jku.at Institute of Computational Perception Agenda Language Modeling with n- grams Recurrent Neural

667 views • 55 slides
Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

444 views • 27 slides
Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides
Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi Convolutional Neural Networks (CNNs) Fully connected (dense) layers have no awareness of spatial information Key concept behind convolutional layers is

983 views • 41 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly used for what sort of task? a) Recognizing images b) Transcribing speech c) Translating documents d) Playing games Neural network review

431 views • 13 slides
Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs Convolutions Pooling, Padding and Stride Convolutional Neural Networks (LeNet) Deep ConvNets (AlexNet) Networks using Blocks (VGG)

1.91k views • 89 slides

More recommend