beyond data and model parallelism for deep neural networks
play

Beyond Data and Model Parallelism for Deep Neural Networks ZHIHAO - PowerPoint PPT Presentation

Sep 08, 2023 •163 likes •284 views

Beyond Data and Model Parallelism for Deep Neural Networks ZHIHAO JIA, MATEI ZAHARIA, ALEX AIKEN SYSML 2019 PRESENTED BY JULIUS LISCHEID Existing Parallelisation Approaches (1/2) DATA PARALLELISM MODEL PARALLELISM Replica of neural

  1. Beyond Data and Model Parallelism for Deep Neural Networks ZHIHAO JIA, MATEI ZAHARIA, ALEX AIKEN SYSML 2019 PRESENTED BY JULIUS LISCHEID

  2. Existing Parallelisation Approaches (1/2) DATA PARALLELISM MODEL PARALLELISM • Replica of neural network on each device • Disjoint subsets of neural network assigned to devices • Each device processes subset of training data • No parameter synchronisation, but requires • After each iteration, parameters are data transfers between operations synchronised • Works well for compute-heavy operations with few parameters (e.g. convolutions)

  3. Existing Parallelisation Approaches (2/2) EXPERT-DESIGNED STRATEGIES AUTOMATED FRAMEWORKS • A. Krizhevsky. One weird trick for parallelizing • A. Mirhoseini et al. Device Placement convolutional neural networks. CoRR 2014. Optimization with Reinforcement Learning. ICML 2017. • Data parallelism for convolutional layers, model • Reinforment learning for model parallelism parallelism for fully-connected layers • Y. Wu et al. Google’s neural machine translation • Z. Jia et al. Exploring hidden dimensions in system: bridging the gap between human and parallelizing convolutional neural networks. CoRR machine translation. CoRR 2016. 2018. • Data parallelism for compute nodes, model • Dynamic Programming for parallelisation of DNNs parallelism for intra-node computation with linear computation graphs • D. Narayanan et al. PipeDream: generalized pipeline parallelism for DNN training. SOSP 2019. • …

  4. The SOAP Search Space Samples (data parallelism) Operators (model parallelism) Attributes (e.g. pixels) Parameters (≈model parallelism)

  5. Hybrid Parallelism in SOAP Example parallelization strategies for 1D convolution

  6. FlexFlow • Trying out strategies on hardware is expensive due to long iteration times • Execution Optimizer uses simulator instead • Measures operator runtime on hardware • Estimates runtime of parallelisation strategies • Delta simulation algorithm uses incremental updates for acceleration • Execution optimizer explores search space with Markov Chain Monte Carlo algorithm

  7. Evaluation (1/2)

  8. Evaluation (2/2)

  9. Review (1/2) STRENGTHS/AGREEMENTS WEAKNESSES/DISAGREEMENTS • Expands search space for parallelisation • Unclear how much SOAP and execution strategies optimiser contribute to training acceleration • Proposes a way to efficiently explore that • Usefulness of Attribute dimension is search space questionable • Leads to an actual speed-up • More end-to-end performance benchmarks would have been useful

  10. Review (2/2) KEY TAKEAWAYS POTENTIAL IMPACT • Training performance of parallelisation • Usage of other search algorithms to explore strategies can be efficiently and accurately parallelisation search space in simulation predicted • Combination of parallelisation search space • The resulting speed-up allows for the with computation graph substitutions exploration of a wider search space (compare Tim’s presentation next week)

  11. Questions? P A S O

  12. Image Citations Images with beige background retrieved from Jia Zhihao’s SysML 19 talk: https://www.youtube.com/watch?v=81l6kkV-OkE All other images extracted from Z. Jia, M. Zaharia, and A. Aiken: Beyond Data and Model Parallelism for Deep Neural Networks, SYSML, 2019.

Recommend

Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia, Matei Zaharia and Alex

Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia, Matei Zaharia and Alex

Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia, Matei Zaharia and Alex Aiken Cristian (cb2015@cam.ac.uk) Content Types of parallelism The SOAP space FlexFlow Evaluation of FlexFlow Critique Types of

420 views • 18 slides
Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2 1 University of Oxford, 2 EPFL ICLR 2020 Workshop on PML4DC Motivation Compressing neural networks can save a large amount of memory and

570 views • 21 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep

AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep

AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep Ll. Berral @BSC Intel Academic Education Mindshare Initiative for AI Introduction Neural networks attempt to imitate the brain neuron

339 views • 21 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long Short-Term Memory: LSTM

366 views • 18 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.37k views • 134 slides
WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

OPTICAL FLOW ESTIMATION WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1. Optical Flow Motivation 2. Neural Networks Basics 3. Optical Flow with Deep Neural Networks 1. PWC-Net Model 2. PWC-Net Results

481 views • 20 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Learning from Limited Data The University of Tokyo / RIKEN AIP Tatsuya Harada Deep Neural

Learning from Limited Data The University of Tokyo / RIKEN AIP Tatsuya Harada Deep Neural

GTC March 18, 2019 Learning from Limited Data The University of Tokyo / RIKEN AIP Tatsuya Harada Deep Neural Networks for Visual Recognition Deep Neural Networks Applications A yellow train on the tracks near a train station. cellphone cup

661 views • 53 slides
arXiv:1303.5778v1 [cs.NE] 22 Mar 2013 Recurrent neural networks (RNNs) are a powerful model for

arXiv:1303.5778v1 [cs.NE] 22 Mar 2013 Recurrent neural networks (RNNs) are a powerful model for

SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton Department of Computer Science, University of Toronto ABSTRACT RNNs are inherently deep in time, since their hidden state is a function

362 views • 5 slides
1 Real Neural Learning Artificial Neuron Model Model network as a graph with cells as nodes

1 Real Neural Learning Artificial Neuron Model Model network as a graph with cells as nodes

Neural Networks Analogy to biological neural systems, the most robust learning systems we know. Attempt to understand natural biological systems CS 391L: Machine Learning through computational modeling. Massive parallelism allows

92 views • 6 slides
CS5242 Neural Networks and Deep Learning Lecture 09: RNN Applications II Wei WANG TA: Yao SHU,

CS5242 Neural Networks and Deep Learning Lecture 09: RNN Applications II Wei WANG TA: Yao SHU,

CS5242 Neural Networks and Deep Learning Lecture 09: RNN Applications II Wei WANG TA: Yao SHU, Juncheng LIU, Qingpeng Cai cs5242@comp.nus.edu.sg Recap Language modelling Training Model the joint probability Model the conditional

1.29k views • 32 slides
Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane Mallat, Ivan Dokmanic, Martin de Hoop cole Normale Suprieure www.di.ens.fr/data Deep Convolutional Networks ( x ) x ( u ) L j L 1 f M (

520 views • 47 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.58k views • 134 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2020 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2020 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2020 1 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.89k views • 143 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides

More recommend