Investigating scalability of recurrent network using dynamic - PowerPoint PPT Presentation

Feb 05, 2024 •418 likes •520 views

Investigating scalability of recurrent network using dynamic batching in PyTorch Devin Taylor November 27, 2018 Computer Laboratory, University of Cambridge, United Kingdom What is dynamic batching? (a) Initial computational graph (b) Batched

Investigating scalability of recurrent network using dynamic batching in PyTorch Devin Taylor November 27, 2018 Computer Laboratory, University of Cambridge, United Kingdom
What is dynamic batching? (a) Initial computational graph (b) Batched computational graph Figure 1: Dynamic batching for a single parse tree [1] 1
TensorFlow Fold • Simplified API for adding dynamic batching to TensorFlow • Last commit on 31 October 2017 (but not depricated) - TensorFlow Eager prioritised [2] • Poor evaluation when compared to TensorFlow • Very little insight into whether dynamic batching is actually useful • Only evaluated on binary trees • All trees had the same shape and size • “Best case scenario” • Inference timing results excluded time to construct static computation graph 2
Dynamic batching in PyTorch • Want to evaluate if concept of dynamic batching is more efficient • PyTorch dynamic computation graphs support direct batching of variable inputs • Can test on real data • Reconduct experiments from Looks et al. (2017) for PyTorch [3] • Implementation already exists - TorchFold [4] • Last commit on 7 July 2018 • No support for PyTorch 0.4+ 3
Evaluation • Sentiment classification with TreeLSTM network [5] • Direct batching • Dynamic batching • Measure inference time for variable batch sizes • Compare to results obtained using TensorFlow Fold • Investigate implementing in additional frameworks for further comparisons • TensorFlow Eager • Knet (Julia) 4
Workplan 7 Dec Gather results and write report 4 20 Dec 17 Dec Investigate implementations in other frameworks 5 14 Dec 10 Dec Rewrite TorchFold for PyTorch 0.4+ and rerun experiment 4 4 Dec Start date Rebuild experiment from Looks et al. (2017) in PyTorch 4 29 Nov 26 Nov Pre-reading 3 23 Nov 21 Nov Task Days End date 5
References i Announcing tensorflow fold: Deep learning with dynamic computation graphs. https://ai.googleblog.com/2017/02/ announcing-tensorflow-fold-deep.html , Feb 2017. Eager execution: An imperative, define-by-run interface to tensorflow. https://research.googleblog.com/2017/10/ eager-execution-imperative-define-by.html , Oct 2017. Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. Deep learning with dynamic computation graphs. arXiv preprint arXiv:1702.02181 , 2017. 6
References ii torchfold. https://github.com/nearai/torchfold , Sep 2017. Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 , 2015. 7

Recommend

Investigating Relational Recurrent Neural Networks with Variable Length Memory Pointer Mahtab

Investigating Relational Recurrent Neural Networks with Variable Length Memory Pointer Mahtab Ahmed and Robert E. Mercer Department of Computer Science University of Western Ontario London, ON, Canada Introduction Memory based Neural

595 views • 13 slides

Single Layer Recurrent Network Bidirectional Symmetric Connection Binary /

Hopfield Network Single Layer Recurrent Network Bidirectional Symmetric Connection Binary / Continuous Units Associative Memory Optimization Problem Hopfield Model Discrete Case Recurrent neural network that uses

288 views • 8 slides

! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary /

Hopfield Network ! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary / Continuous Units ! Associative Memory ! Optimization Problem Hopfield Model Discrete Case Recurrent neural network that

733 views • 51 slides

Recurrent Neural Network Agenda Recurrent Neural Network

Recurrent Neural Network Agenda Recurrent Neural Network NER sequence tagging RNN

670 views • 46 slides

UArctic Thematic Network on Herbivory An international network of researchers investigating

Proposal for a new UArctic Thematic Network on Herbivory An international network of researchers investigating the role of herbivores in arctic and alpine ecosystems Established in 2014, currently involves more than 200 scientists from

171 views • 5 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

Contents Introduction Scalability and Accuracy in a Architecture Large-Scale Network

Contents Introduction Scalability and Accuracy in a Architecture Large-Scale Network Emulator Implementation and Evaluation Accuracy vs. Scalability Tradeoff Nov. 12, 2003 Case Studies Byung-Gon Chun 1 2 Motivation

299 views • 5 slides

Scalability and Replication Marco Serafini COMPSCI 532 Lecture 13 Scalability 2 Scalability

Scalability and Replication Marco Serafini COMPSCI 532 Lecture 13 Scalability 2 Scalability Ideal world Linear scalability Speedup Reality Ideal Bottlenecks For example: central coordinator When do we stop

938 views • 36 slides

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views • 55 slides

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides

Recurrent Neural Network Grammars NAACL-HLT 2016 Authors: Chris Dyer, Adhiguna Kuncoro, Miguel

Recurrent Neural Network Grammars NAACL-HLT 2016 Authors: Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, Noah A. Smith Presenter: Che-Lin Huang Motivation Sequential recurrent neural networks (RNNs) are remarkably effective models of

516 views • 17 slides

Constraint satisfaction Unfolding a recurrent network into a feedforward one t=3 t=2 t=1

Constraint satisfaction Unfolding a recurrent network into a feedforward one t=3 t=2 t=1 Units represent hypotheses about parts of a problem Weights code constraints on how hypotheses can combine (i.e., the degree to which they are

690 views • 3 slides

Recurrent Neural Network Xiaogang Wang xgwang@ee.cuhk.edu.hk February 26, 2019 cuhk Xiaogang

Recurrent Neural Network Xiaogang Wang xgwang@ee.cuhk.edu.hk February 26, 2019 cuhk Xiaogang Wang (CUHK) Recurrent Neural Network February 26, 2019 1 / 52 Outline 1 Recurrent neural networks Recurrent neural networks BP on RNN Variants

1.22k views • 52 slides

Poor Man's Social Network Consistently Trade Freshness For Scalability Zhiwu Xie, Jinyang Liu,

Poor Man's Social Network Consistently Trade Freshness For Scalability Zhiwu Xie, Jinyang Liu, Herbert Van de Sompel, Johann van Reenen and Ramiro Jordan Outline Scaling feed following Algorithm Experiment and results Conclusions

501 views • 24 slides

Scalability of Future Mobile Networks Through Network Independent Application Layer Mobility and

Scalability of Future Mobile Networks Through Network Independent Application Layer Mobility and Context Monitoring Modeling of Adaptive Systems htups://www.mas.wiwi.uni-due.de/en Florian Metzger 2015/12/10 Motivation QoS and QoE of TCP

494 views • 15 slides

Product2Vec : MRNe Net-Pr A A Multi ti-task Recurrent Ne Neural Ne Network for Product

Product2Vec : MRNe Net-Pr A A Multi ti-task Recurrent Ne Neural Ne Network for Product Embedding Embeddings Arijit Biswas, Mukul Bhutani and Subhajit Sanyal Machine Learning, Amazon, Bangalore, India {barijit,mbhutani,subhajs}@amazon.com

731 views • 21 slides

Investigations on Phrase-based Decoding with Recurrent Neural Network Language and Translation

Investigations on Phrase-based Decoding with Recurrent Neural Network Language and Translation Models Tamer Alkhouli, Felix Rietig, and Hermann Ney surname@cs.rwth-aachen.de Tenth Workshop on Statistical Machine Translation Lisbon, Portugal

563 views • 24 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views • 21 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

583 views • 47 slides

Non-projective Dependency-based Pre-Reordering with Recurrent Neural Network for Machine

Motivation Non-projective non-tree-local pre-reordering Reordering automaton GRU-RM Conclusions Non-projective Dependency-based Pre-Reordering with Recurrent Neural Network for Machine Translation Antonio Valerio Miceli Barone Giuseppe

1.57k views • 80 slides

Auto-associator Recurrent network (settles over time) Same units serve as input and output 1 /

Auto-associator Recurrent network (settles over time) Same units serve as input and output 1 / 24 3 / 24 Representing both general and specific knowledge Distributed memory model (McClelland & Rumelhart, 1985) Delta-rule learning in an

482 views • 6 slides

What do Recurrent Neural Network Grammars Learn About Syntax ? Authors: Adhiguna Kuncoro, Miguel

What do Recurrent Neural Network Grammars Learn About Syntax ? Authors: Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, Noah A. Smith Presented by: Triveni Putti Paper link: https://arxiv.org/pdf/1611.05774.pdf

508 views • 23 slides

On The Scalability of Storage Sub-System Back-end Network Yan Li, Roland Ibbett, Nigel Topham and

V I N E U R S E I H T Y T O H F G R E U D B I N On The Scalability of Storage Sub-System Back-end Network Yan Li, Roland Ibbett, Nigel Topham and Tim Courtney School of Informatics University of Edinburgh Xyratex,

387 views • 12 slides

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides