Recurrent Neural Networks Sharan Narang May 9, 2017 Silicon Valley - PowerPoint PPT Presentation

Oct 19, 2023 •409 likes •603 views

Exploring Sparsity in Recurrent Neural Networks Sharan Narang May 9, 2017 Silicon Valley AI Lab Speech Recognition with Deep Learning English Scaling with Data Compa paris rison on of Spe peech Recogni gnition tion App pproac roaches

Exploring Sparsity in Recurrent Neural Networks Sharan Narang May 9, 2017 Silicon Valley AI Lab
Speech Recognition with Deep Learning English
Scaling with Data Compa paris rison on of Spe peech Recogni gnition tion App pproac roaches hes Title Accuracy Deep Learning Traditional methods Data + Model Size (Speed)
Model Sizes 500.00 461.87 400.00 300.00 270.79 Number of Parameters (in millions) 200.00 Size (in MB) 115.47 100.00 67.70 32.56 8.14 0.00 Deep Speech 1 Deep Speech 2 Deep Speech 2 (RNN) (GRU) Baidu du Speec ech h Models els
Future Vision
Sparse Neural Networks
Pruning Weights Dense Initial Network Pruning Weights Sparse Final Network Start of Training During Training End of Training Epochs
Pruning Approach 0.5 0.4 Prune Threshold 0.3 Recurrent 0.2 Linear 0.1 0 0 5 10 15 20 Epoch
Pruning Layers 100% 95% 90% Sparsity 85% Pruned Percent 80% 75% 70% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Layers
Results Model Layer Size # of Params CER Relative Perf RNN Dense 1760 67 million 10.67 0.0% RNN Sparse 1760 8.3 million 12.88 -20.71% RNN Sparse 2560 11.1 million 10.59 0.75% RNN Sparse 3072 16.7 million 10.25 3.95% GRU Dense 2560 115 million 9.55 0.0% GRU Sparse 2560 13 million 10.87 -13.82% GRU Sparse 3568 17.8 million 9.76 -2.2%
Equal Parameter Networks 60 small_dense_train small_dense_dev0 55 large_sparse_train large_sparse_dev0 50 45 CTC Cost 40 35 30 25 20 0 5 10 15 20 Epoch Number
Sparsity v/s Accuracy 10% 10.89 CER Baseline line 0% -10% 13.0 CER Relative Accuracy -20% -30% -40% -50% -60% 17.4 CER -70% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Sparsity
Models don’t need to be retrained
Compression RNN Model 10 8.11 8 Compression 6.06 6 4.04 4 2 0 1760 Sparse 2560 Sparse 3072 Sparse Sparse se Models els
Speedup 12 10 10 8 6 5.33 Measured Speedup 3.89 Expected Speedup 4 2.90 1.93 2 1.16 0 1760 Sparse 2560 Sparse 3072 Sparse Sparse se Models els
Conclusion • Sparse Neural Networks can achieve good accuracy while significantly reducing the number of parameters • Threshold based approach works for fully connected layers, recurrent layers and GRU layers • Improvements in Sparse Matrix Vector libraries can result in higher speedup for Sparse Neural Networks
Thank You!
Sharan Narang sharan@baidu.com http://research.baidu.com Silicon Valley AI Lab

Recommend

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views • 27 slides

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views • 21 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

583 views • 47 slides

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer University of Washington [most slides from Yejin Choi] RECURRENT NEURAL L NE NETWOR WORKS Recurrent Neural Networks (RNNs) Each input

787 views • 29 slides

Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Sequential Data with Neural Networks Recurrent Neural

303 views • 4 slides

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0 L0 M*V Processor N N N N N N Project Brainwave Pretrained DNN Model Scalable DNN Hardware Neural Processing Unit Microservice Extract

525 views • 30 slides

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova March 9, 2020 Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/

538 views • 18 slides

Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar

Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar ltekin University of Tbingen Seminar fr Sprachwissenschaft July 5, 2016 Neural networks: a quick summary Recurrent neural networks July 5, 2016

570 views • 35 slides

The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente

The Power of Linear Recurrent The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente neuronale Netze? Frieder Stolzenburg Overview Frieder Stolzenburg Introduction Recurrent Neural Networks

1.17k views • 66 slides

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent Neural Networks Compact schematic drawing of standard multi-layer perceptron (MLP) output hidden input Modeling sequential data So far we

796 views • 50 slides

INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan

INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan Oepen University of Oslo November 6, 2018 Most Recently: Variants of Recurrent Neural Networks To a first approximation, the de facto consensus in

339 views • 23 slides

RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

Outline Recurrent Neural Networks (RNNs) Neural Machine Transla/on NMT basics (Sutskever et al., 2014) ABenCon mechanism (Bahdanau et al., 2015) Marcello Federico 2016 Recurrent Neural Networks (RNNs) Based on slides kindly provided

437 views • 24 slides

Recurrent Neural Networks LING572 Advanced Statistical Methods for NLP March 5 2020 1 Outline

Recurrent Neural Networks LING572 Advanced Statistical Methods for NLP March 5 2020 1 Outline Word representations and MLPs for NLP tasks Recurrent neural networks for sequences Fancier RNNs Vanishing/exploding gradients

921 views • 91 slides

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap from last time Feed-Forward Neural Network: Multilayer Perceptron + 0 ) = (

1.04k views • 93 slides

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO

Recurrent Neural Networks (RNNs) for NLP MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO 29-05-2017 TABLE OF CONTENTS DEEP LEARNING FOR NLP WORD EMBEDDINGS RECURRENT NEURAL NETWORKS (RNNs) APPLICATION UPSKILLING 2 ZALANDO Our

712 views • 23 slides