gated orthogonal recurrent units on learning to forget
play

Gated Orthogonal Recurrent Units: On Learning to Forget Li Jing, a - PowerPoint PPT Presentation

Oct 26, 2022 •226 likes •321 views

Gated Orthogonal Recurrent Units: On Learning to Forget Li Jing, a lar Glehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Solja i , Yoshua Bengio Gradient Vanishing/Explosion Problem During backpropagation through time,

  1. Gated Orthogonal Recurrent Units: On Learning to Forget Li Jing, Ça ğ lar Gülçehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Solja č i ć , Yoshua Bengio

  2. Gradient Vanishing/Explosion Problem • During backpropagation through time, hidden to hidden Jacobian matrix is multiplied multiple times. • Gradient vanishing/explosion makes RNN hard to train Li Jing

  3. Conventional Solution: LSTM • Practically, gradient clipping is required • slow to learn long term dependency Li Jing

  4. Unitary/Orthogonal RNN Unitary/Orthogonal matrices keep the norm of vectors: By enforcing hidden to hidden transition matrix to be unitary/ orthogonal, no matter how many time steps are propagated, the norm of the gradient will stay the same • Restricted-capacity Unitary Matrix Parametrization (Arjovsky, ICML 2016) • Full-capacity Unitary Matrix by projection (Wisdom, NIPS 2016) • Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN (Jing, ICML 2017) • Orthogonal Matrix Parametrization by reflection (Mhammedi, ICML 2017) • Orthogonal Matrix by regularization (Vorontsov, ICML 2017) Li Jing

  5. Limitation for basic Orthogonal RNN • No forgetting mechanism • Limited Memory size

  6. Applying Gated System to Orthogonal RNN z 1-z modReLU r W x U IN h OUT Gated Orthogonal Recurrent Unit Unitary/Orthogonal Matrices Long Term Dependency Gated Mechanism Forgetting

  7. Experiment results Synthetic Tasks: GORU is the only one succeeding in all tasks • Parenthesis Task • Copying Task • Denoise Task Li Jing

  8. Experiment results Real Tasks: GORU outperforms • Question Answering Task all other models • Speech Task Li Jing

  9. Thank you

Recommend

CS7015 (Deep Learning) : Lecture 15 Long Short Term Memory Cells (LSTMs), Gated Recurrent Units

CS7015 (Deep Learning) : Lecture 15 Long Short Term Memory Cells (LSTMs), Gated Recurrent Units

CS7015 (Deep Learning) : Lecture 15 Long Short Term Memory Cells (LSTMs), Gated Recurrent Units (GRUs) Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/43 Mitesh M. Khapra CS7015 (Deep

467 views • 43 slides
CS7015 (Deep Learning) : Lecture 15 Long Short Term Memory Cells (LSTMs), Gated Recurrent Units

CS7015 (Deep Learning) : Lecture 15 Long Short Term Memory Cells (LSTMs), Gated Recurrent Units

CS7015 (Deep Learning) : Lecture 15 Long Short Term Memory Cells (LSTMs), Gated Recurrent Units (GRUs) Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/43 Mitesh M. Khapra CS7015 (Deep

4.9k views • 196 slides
Outline Gated Feedback Recurrent Neural Networks. arXiv1502. Introduction: RNN & Gated RNN

Outline Gated Feedback Recurrent Neural Networks. arXiv1502. Introduction: RNN & Gated RNN

Outline GF-RNN ReNet Outline Gated Feedback Recurrent Neural Networks. arXiv1502. Introduction: RNN & Gated RNN Gated Feedback Recurrent Neural Networks (GF-RNN) Experiments: Character-level Language Modeling & Python Program

631 views • 26 slides
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification Duyu Tang,

Document Modeling with Gated Recurrent Neural Network for Sentiment Classification Duyu Tang,

Document Modeling with Gated Recurrent Neural Network for Sentiment Classification Duyu Tang, Bing Qin, Ting Liu Harbin Institute of Technology 1 Sentiment Classification Given a piece of text, sentiment classification focus on inferring

287 views • 27 slides
Auto-associator Recurrent network (settles over time) Same units serve as input and output 1 /

Auto-associator Recurrent network (settles over time) Same units serve as input and output 1 /

Auto-associator Recurrent network (settles over time) Same units serve as input and output 1 / 24 3 / 24 Representing both general and specific knowledge Distributed memory model (McClelland & Rumelhart, 1985) Delta-rule learning in an

482 views • 6 slides
CHAPTER VII VII CHAPTER Learning in Recurrent Networks Learning in Recurrent Networks CHAPTER

CHAPTER VII VII CHAPTER Learning in Recurrent Networks Learning in Recurrent Networks CHAPTER

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER VII VII CHAPTER Learning in Recurrent Networks Learning in Recurrent Networks CHAPTER VI : VI : Learning in CHAPTER Learning in Recurrent Recurrent Networks Networks Introduction We

464 views • 17 slides
On the use of phone-gram units in recurrent neural networks for language identification

On the use of phone-gram units in recurrent neural networks for language identification

On the use of phone-gram units in recurrent neural networks for language identification Christian Salamea, Luis F. D'Haro, Ricardo de Crdoba, Rubn San-Segundo Speech Technology Group. Dept. of Electronic Engineering. Universidad

166 views • 4 slides
Gated Path Planning Networks Lisa Lee Machine Learning Department Carnegie Mellon University

Gated Path Planning Networks Lisa Lee Machine Learning Department Carnegie Mellon University

Gated Path Planning Networks Lisa Lee Machine Learning Department Carnegie Mellon University Joint work with Emilio Parisotto, Devendra Chaplot, Eric Xing, & Ruslan Salakhutdinov ICML 2018 Path Planning Gated Path Planning Networks

280 views • 26 slides
Classification of self-orthogonal F q + u F q -codes Classification of self-orthogonal F q + u F q

Classification of self-orthogonal F q + u F q -codes Classification of self-orthogonal F q + u F q

Self-orthogonal Codes over F q + u F q Rowena Alma Betty, Fidel Nemenzo and Lucky Erap Galvez University of the Philippines-Diliman LAWCI 2018 Classification of self-orthogonal F q + u F q -codes Classification of self-orthogonal F q + u F q

975 views • 59 slides
Range gated cameras technology and its applications Friday, October 10, 14 Range gated cameras

Range gated cameras technology and its applications Friday, October 10, 14 Range gated cameras

Range gated cameras technology and its applications Friday, October 10, 14 Range gated cameras can have several names * Time resolved cameras * Range gated cameras * Laser tomography * Grindad laserkamera / avbildning ( SV) All systems

647 views • 25 slides
Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides
Legendre Memory Units (LMUs) Continuous-Time Representation in Recurrent Neural Networks Aaron

Legendre Memory Units (LMUs) Continuous-Time Representation in Recurrent Neural Networks Aaron

Legendre Memory Units (LMUs) Continuous-Time Representation in Recurrent Neural Networks Aaron Voelker, Ivana Kaji, Chris Eliasmith Image caption Music composition generation LSTMs Speech recognition Disease prediction Machine

482 views • 15 slides
1 Mammalian Neurons Have Several Types of Voltage-Gated Ion Channels Why do neurons need so many

1 Mammalian Neurons Have Several Types of Voltage-Gated Ion Channels Why do neurons need so many

Voltage-Gated Ion Channels in Health and Disease jdk3 Principles of Neural Science, chapter 9 Voltage-Gated Ion Channels in Health and Disease I. Multiple functions of voltage- gated ion channels II. Neurological diseases involving

613 views • 15 slides
Orthogonal Complements and Orthonormal Matrices Orthogonal Complements Defn. For a set W , the

Orthogonal Complements and Orthonormal Matrices Orthogonal Complements Defn. For a set W , the

Orthogonal Complements and Orthonormal Matrices Orthogonal Complements Defn. For a set W , the orthogonal comple- ment denoted W is the set of all vectors that are orthogonal to all of W . orthoTWO: 2 Orthogonal Complements are Subspaces

590 views • 13 slides
Learning multisensory integration with stochastic variational learning in recurrent spiking

Learning multisensory integration with stochastic variational learning in recurrent spiking

Learning multisensory integration with stochastic variational learning in recurrent spiking networks Presented by : Sharbatanu Chatterjee 1 , 2 Guided by : Aditya Gilra 1 & Johanni Brea 1 1 Laboratory of Computational Neuroscience, EPFL,

456 views • 29 slides
Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO 29-05-2017 TABLE OF CONTENTS DEEP LEARNING FOR NLP WORD EMBEDDINGS RECURRENT NEURAL NETWORKS (RNNs) APPLICATION UPSKILLING 2 ZALANDO Our

712 views • 23 slides
! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary /

! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary /

Hopfield Network ! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary / Continuous Units ! Associative Memory ! Optimization Problem Hopfield Model Discrete Case Recurrent neural network that

733 views • 51 slides
Single Layer Recurrent Network Bidirectional Symmetric Connection Binary /

Single Layer Recurrent Network Bidirectional Symmetric Connection Binary /

Hopfield Network Single Layer Recurrent Network Bidirectional Symmetric Connection Binary / Continuous Units Associative Memory Optimization Problem Hopfield Model Discrete Case Recurrent neural network that uses

288 views • 8 slides
Phonics Workshop for Parents Learning to read Sometimes as adults we forget how difficult

Phonics Workshop for Parents Learning to read Sometimes as adults we forget how difficult

Phonics Workshop for Parents Learning to read Sometimes as adults we forget how difficult learning to read is. When first learning to read letters are just a series of symbols. Children need lots of practise and repetition to learn the names

467 views • 18 slides
1 Early Computers Were Made of Thousands of Identical Electronic Components II. Fine Control of

1 Early Computers Were Made of Thousands of Identical Electronic Components II. Fine Control of

Voltage-Gated Ion Channels in Voltage-Gated Ion Channels in Health and Disease Health and Disease jdk3 I. Multiple functions of voltage- gated ion channels Principles of Neural Science, chapter 9 II. Neurological diseases involving

259 views • 8 slides
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group Mario Lezcano-Casado David Martnez-Rubio Mathematical Institute Department of Computer Science June 12, 2019 Cheap Orthogonal

3.28k views • 17 slides
Lecture 11: Recurrent Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman

Lecture 11: Recurrent Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman

Lecture 11: Recurrent Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline Forgetting, remembering and updating (review) Gated networks, LSTM and GRU RNN Structures Bidirectional Deep

833 views • 54 slides
Codes from orbit matrices of weakly q -self-orthogonal 1-designs 2 / 19 On some self-orthogonal

Codes from orbit matrices of weakly q -self-orthogonal 1-designs 2 / 19 On some self-orthogonal

On some self-orthogonal codes from M 11 On some self-orthogonal codes from M 11 Ivona Novak (inovak@math.uniri.hr) joint work with Vedrana Mikuli c Crnkovi c (vmikulic@math.uniri.hr) Department of Mathematics, University of Rijeka Finite

786 views • 19 slides

More recommend