how to construct deep recurrent neural
play

How to Construct Deep Recurrent Neural Networks AUTHORS: R. - PowerPoint PPT Presentation

Oct 04, 2023 โ€ข251 likes โ€ข395 views

How to Construct Deep Recurrent Neural Networks AUTHORS: R. PASCANU, C. GULCEHRE, K. CHO, Y. BENGIO PRESENTATION: HAROUN HABEEB PAPER: HTTPS://ARXIV.ORG/ABS/1312.6026 This presentation Motivation Formal RNN paradigm Deep RNN designs

  1. How to Construct Deep Recurrent Neural Networks AUTHORS: R. PASCANU, C. GULCEHRE, K. CHO, Y. BENGIO PRESENTATION: HAROUN HABEEB PAPER: HTTPS://ARXIV.ORG/ABS/1312.6026

  2. This presentation Motivation Formal RNN paradigm Deep RNN designs Experiments Note on training Takeaways

  3. Motivation: Better RNNs? Depth makes feedforward neural networks more expressive What about RNNS? How do you make them deep? Does depth help?

  4. ๐’Š ๐‘ข = ๐‘” โ„Ž (๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 ) Conventional ๐’› ๐‘ข = ๐‘” ๐‘ ๐’Š ๐‘ข RNNs Specifically: โ„Ž ๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 ; ๐‘ฟ, ๐‘ฝ = ๐œš โ„Ž ๐‘ฟ ๐‘ˆ ๐’Š ๐‘ขโˆ’1 + ๐‘ฝ ๐‘ผ ๐’š ๐‘ข ๐‘” ๐‘ ๐’Š ๐‘ข ; ๐‘พ = ๐œš ๐‘ (๐‘พ ๐‘ˆ ๐’Š ๐‘ข ) ๐‘” โ–ช How general is this? โ–ช How easy is it to represent an LSTM/GRU in this form? โ–ช What about bias terms? โ–ช How would you make an LSTM deep?

  5. THE DEEPENING

  6. ๐’› ๐‘ข = ๐‘” ๐‘ ๐’Š ๐‘ข DT(S)-RNN ๐’Š ๐‘ข = ๐‘” โ„Ž (๐‘• ๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 , ๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 ) Specifically: ๐‘ง ๐‘ข = ๐œ”(๐‘‹โ„Ž ๐‘ข ) โ„Ž ๐‘ข = ๐œš ๐‘€ ( ๐‘ˆ ๐œš ๐‘€โˆ’1 โ€ฆ ๐‘Š ๐‘ˆ ๐œš 1 ๐‘Š ๐‘ˆ โ„Ž ๐‘ขโˆ’1 + ๐‘‰๐‘ฆ ๐‘ข ๐‘Š ๐‘€ 2 1 + เดฅ ๐‘‹ ๐‘ˆ โ„Ž ๐‘ขโˆ’1 +เดฅ ๐‘‰ ๐‘ˆ ๐‘ฆ ๐‘ข )

  7. ๐’› ๐‘ข = ๐‘” ๐‘ ๐’Š ๐‘ข DOT(S)-RNN ๐’Š ๐‘ข = ๐‘” โ„Ž (๐‘• ๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 , ๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 ) Specifically: ๐‘ˆ ๐œ” ๐‘€ (โ€ฆ ๐‘‹ ๐‘ˆ ๐œ” 1 ๐‘‹ ๐‘ˆ โ„Ž ๐‘ข ) ๐‘ง ๐‘ข = ๐œ” 0 (๐‘‹ ๐‘€ 1 โ„Ž ๐‘ข = ๐œš ๐‘€ ( ๐‘ˆ ๐œš ๐‘€โˆ’1 โ€ฆ ๐‘Š ๐‘ˆ ๐œš 1 ๐‘Š ๐‘ˆ โ„Ž ๐‘ขโˆ’1 + ๐‘‰๐‘ฆ ๐‘ข ๐‘Š ๐‘€ 2 1 + เดฅ ๐‘‹ ๐‘ˆ โ„Ž ๐‘ขโˆ’1 +เดฅ ๐‘‰ ๐‘ˆ ๐‘ฆ ๐‘ข )

  8. 0 = ๐’ˆ โ„Ž 0 (๐’š ๐‘ข , ๐’Š ๐‘ขโˆ’1 0 ๐’Š ๐‘ข ) sRNN (๐‘š) = ๐‘” (๐‘š) (๐’Š ๐‘ข ๐‘šโˆ’1 , ๐’Š ๐‘ขโˆ’1 ๐‘š โˆ€๐‘š โˆถ ๐’Š ๐‘ข ) โ„Ž (๐‘€) ๐’› ๐‘ข = ๐‘” ๐‘ ๐’Š ๐‘ข Specifically: (๐‘€) ๐’› ๐‘ข = ๐œ” ๐‘‹ ๐‘ˆ ๐’Š ๐‘ข (0) = ๐œš 0 (0) ๐‘ˆ ๐’š ๐‘ข + ๐‘‹ ๐‘ˆ ๐’Š ๐‘ขโˆ’1 โ„Ž ๐‘ข ๐‘‰ 0 0 ๐‘š = ๐œš ๐‘š ๐‘šโˆ’1 + ๐‘‹ (๐‘š) ๐‘ˆ ๐’Š ๐‘ข ๐‘ˆ ๐’Š ๐‘ขโˆ’1 โˆ€๐‘š: ๐’Š ๐‘ข ๐‘‰ ๐‘š ๐‘š

  9. Experiment 0: Parameter count Food for thought: Not clear which one has most number of parameters โ€“ sRNN or DOT(S)-RNN.

  10. Experiment 1: Polyphonic Music Prediction Next Task: note(s) Sequence of musical notes Food for thought: Sure, depth helps, but * helps a lot more in this case. What about RNN* and other models with *?

  11. Experiment 2: Language Modelling Next Task : Sequence of characters/words character/word (LM on PTB) Food for thought: Deepening LSTMs? Stack them or DOT(S) them?

  12. Note on training โ–ช Training RNNs can be hard because of vanishing/exploding gradients. โ–ช Authors did a bunch of things: โ–ช Clipped gradients, threshold = 1 โ–ช Sparse weight matrices ( ๐‘‹ 0 = 20 ) โ–ช โ‡’ max ๐‘—,๐‘˜ ๐‘‹ ๐‘—,๐‘˜ = 1 Normalized weight matrices โ–ช Add gaussian noise to gradients โ–ช Used dropout, maxout, ๐‘€ ๐‘ž units

  13. Takeaways โ–ช Plain, shallow RNNs are not great. โ–ช DOT-RNNs do well. Following should be deep networks โ–ช ๐‘ง = ๐‘”(โ„Ž, ๐‘ฆ) โ–ช โ„Ž ๐‘ข = ๐‘” ๐‘• ๐‘ฆ ๐‘ข , โ„Ž ๐‘ขโˆ’1 , ๐‘ฆ ๐‘ข , โ„Ž ๐‘ขโˆ’1 - both ๐‘” and ๐‘• โ–ช Training can be really hard. โ–ช Thresholding gradients, Dropout, maxout units are helpful/needed โ–ช LSTMs are good Questions?

Recommend

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views โ€ข 48 slides
CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views โ€ข 21 slides
CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

583 views โ€ข 47 slides
Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views โ€ข 18 slides
arXiv:1303.5778v1 [cs.NE] 22 Mar 2013 Recurrent neural networks (RNNs) are a powerful model for

arXiv:1303.5778v1 [cs.NE] 22 Mar 2013 Recurrent neural networks (RNNs) are a powerful model for

SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton Department of Computer Science, University of Toronto ABSTRACT RNNs are inherently deep in time, since their hidden state is a function

362 views โ€ข 5 slides
Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2. Recurrent neural networks: An abstraction 3. Usage patterns for RNNs 4. BiDirectional RNNs 5. A concrete example: The Elman RNN 6. The vanishing

721 views โ€ข 60 slides
Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2. Recurrent neural networks: An abstraction 3. Usage patterns for RNNs 4. BiDirectional RNNs 5. A concrete example: The Elman RNN 6. The vanishing

521 views โ€ข 33 slides
Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2. Recurrent neural networks: An abstraction 3. Usage patterns for RNNs 4. BiDirectional RNNs 5. A concrete example: The Elman RNN 6. The vanishing

964 views โ€ข 18 slides
Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2.

Recurrent Neural Networks CS 6956: Deep Learning for NLP Overview 1. Modeling sequences 2. Recurrent neural networks: An abstraction 3. Usage patterns for RNNs 4. BiDirectional RNNs 5. A concrete example: The Elman RNN 6. The vanishing

3.07k views โ€ข 30 slides
Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views โ€ข 43 slides
Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks

Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks

Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks Navid Rekab-Saz navid.rekabsaz@jku.at Institute of Computational Perception Agenda Language Modeling with n- grams Recurrent Neural

667 views โ€ข 55 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views โ€ข 34 slides
Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views โ€ข 55 slides
ECE 6504: Deep Learning for Perception Topics: Recurrent Neural Networks (RNNs) BackProp

ECE 6504: Deep Learning for Perception Topics: Recurrent Neural Networks (RNNs) BackProp

ECE 6504: Deep Learning for Perception Topics: Recurrent Neural Networks (RNNs) BackProp Through Time (BPTT) Vanishing / Exploding Gradients [Abhishek:] Lua / Torch Tutorial Dhruv Batra Virginia Tech Administrativia HW3

458 views โ€ข 25 slides
Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long Short-Term Memory: LSTM

366 views โ€ข 18 slides
CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views โ€ข 27 slides
Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The standard DNN/CNN paradigms (x,y) - ordered pair of data vectors/images (x) and target (y) Moving to sequence data (x(t),y(t)) where this

780 views โ€ข 38 slides
Natural Language Video Description using Deep Recurrent Neural Networks Thesis Proposal 23 Nov.

Natural Language Video Description using Deep Recurrent Neural Networks Thesis Proposal 23 Nov.

Natural Language Video Description using Deep Recurrent Neural Networks Thesis Proposal 23 Nov. 2015 Subhashini Venugopalan University of Texas at Austin 1 Problem Statement Generate descriptions for events depicted in video clips A monkey

1.2k views โ€ข 62 slides
Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO 29-05-2017 TABLE OF CONTENTS DEEP LEARNING FOR NLP WORD EMBEDDINGS RECURRENT NEURAL NETWORKS (RNNs) APPLICATION UPSKILLING 2 ZALANDO Our

712 views โ€ข 23 slides
Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views โ€ข 105 slides
Natural-Language Video Description with Deep Recurrent Neural Networks June 2017 Subhashini

Natural-Language Video Description with Deep Recurrent Neural Networks June 2017 Subhashini

Natural-Language Video Description with Deep Recurrent Neural Networks June 2017 Subhashini Venugopalan University of Texas at Austin 1 Problem Statement Generate descriptions for events depicted in video clips A monkey pulls a dogs tail

1.11k views โ€ข 94 slides
CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Recurrent neural networks Long short term memory (LSTM) networks Deep

116 views โ€ข 11 slides
Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views โ€ข 27 slides
EE-559 Deep learning 11. Recurrent Neural Networks and Natural Language Processing Fran

EE-559 Deep learning 11. Recurrent Neural Networks and Natural Language Processing Fran

EE-559 Deep learning 11. Recurrent Neural Networks and Natural Language Processing Fran cois Fleuret https://fleuret.org/dlc/ June 16, 2018 COLE POLYTECHNIQUE FDRALE DE LAUSANNE Inference from sequences Fran cois Fleuret

1.3k views โ€ข 104 slides

More recommend