parallelized training of deep nn
play

Parallelized Training of Deep NN Comparison of Current Concepts and - PowerPoint PPT Presentation

Nov 30, 2022 •357 likes •515 views

Parallelized Training of Deep NN Comparison of Current Concepts and Frameworks Sebastian Jger, Hans-Peter Zorn, Stefan Igel, Christian Zirpins Rennes, Dec 10, 2018 Motivation Need to scale the training of neural networks horizontally

  1. Parallelized Training of Deep NN Comparison of Current Concepts and Frameworks Sebastian Jäger, Hans-Peter Zorn, Stefan Igel, Christian Zirpins Rennes, Dec 10, 2018

  2. Motivation › Need to scale the training of neural networks horizontally › Kubernetes based technology stack › Scalability of concepts and frameworks 2

  3. Distributed Training Methods Data Parallelism 3

  4. Data Parallelism Centralized Parameter Server TensorFlow: https://www.tensorflow.org 4

  5. Data Parallelism Decentralized Parameter Server Apache MXNet: http://mxnet.apache.org 5

  6. Experimental Setup Environment › Google Kubernetes Engine › CPU: 2.6 GHz › Ubuntu 16.04 › TensorFlow 1.8.0 › MXNet 1.3.0 6

  7. Experimental Setup Networks Convolutional NN Recurrent NN › LeNet-5 › LSTM › 5 layer › 2 layer › 10 classes › 200 units › Fashion MNIST › Penn Tree Bank › 28x28 gray-scale › 1.000.000 words 7

  8. Experimental Setup Metrics 8

  9. Results Convolutional Neural Network 9

  10. Results Convolutional Neural Network 10

  11. Results Recurrent Neural Network 11

  12. Summarizating the Experiments Decentralized Parameter Server ... › more robust regarding increasing communication effort › scales better for small NN For bigger/ more complex NN … › no significant difference between concepts 12

  13. Conclusion MXNet ... › for small NN better scalability and throughput › for bigger NN higher throughput › less and less complicated code › easier to scale up training 13

  14. Thank you Sebastian Jäger @se_jaeger inovex GmbH Ludwig-Erhard-Allee 6 76131 Karlsruhe sebastian.jaeger@inovex.de

Recommend

Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit

Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit activation) Parametric Gradient-based Learning Backward Non-parametric (gradient

650 views • 21 slides
BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding ( B idirectional

BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding ( B idirectional

BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding ( B idirectional E ncoder R epresentations from T ransformers) Jacob Devlin Google AI Language Pre-training in NLP Word embeddings are the basis of deep

1.56k views • 37 slides
FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides
Speeding R up on your computer by parallelized computations a geostatistical case study

Speeding R up on your computer by parallelized computations a geostatistical case study

Speeding R up on your computer by parallelized computations a geostatistical case study Andreas Papritz Department of Environmental Systems Science, ETH Zurich Statistik Stadt Zrich papritz@env.ethz.ch outline motivating example:

368 views • 34 slides
Normalization Techniques in Training of Deep Neural Networks Lei Huang ( ) State Key

Normalization Techniques in Training of Deep Neural Networks Lei Huang ( ) State Key

Normalization Techniques in Training of Deep Neural Networks Lei Huang ( ) State Key Laboratory of Software Development Environment, Beihang University Mail:huanglei@nlsde.buaa.edu.cn August 17 th , 2017 Outline Introduction to Deep

845 views • 47 slides
Parallelized and Vectorized Tracking Using Kalman Filter with CMS Detector Geometry and Events G.

Parallelized and Vectorized Tracking Using Kalman Filter with CMS Detector Geometry and Events G.

Parallelized and Vectorized Tracking Using Kalman Filter with CMS Detector Geometry and Events G. Cerati 4 , P. Elmer 3 , M. Kortelainen 4 , S. Krutelyov 1 , S. Lantz 2 , M. Lefebvre 3 , M. Masciovecchio 1 , K. McDermott 2 , B. Norris 5 , D. Riley

390 views • 23 slides
Optimization for Training Deep Models Xiaogang Wang xgwang@ee.cuhk.edu.hk February 12, 2019

Optimization for Training Deep Models Xiaogang Wang xgwang@ee.cuhk.edu.hk February 12, 2019

Optimization Basics Optimization of training deep neural networks Multi-GPU Training Optimization for Training Deep Models Xiaogang Wang xgwang@ee.cuhk.edu.hk February 12, 2019 cuhk Xiaogang Wang Optimization for Training Deep Models

1.69k views • 72 slides
Parallelized RSA Algorithm: An Analysis With Performance Evaluation using OpenMP Library in High

Parallelized RSA Algorithm: An Analysis With Performance Evaluation using OpenMP Library in High

Parallelized RSA Algorithm: An Analysis With Performance Evaluation using OpenMP Library in High Performance Computing Environment Md. Ahsan Ayub , Zishan Ahmed Onik, Steven Smith Department of Computer Science Tennessee Technological

507 views • 19 slides
affyPara: Parallelized preprocessing algorithms for high-density oligonucleotide array data

affyPara: Parallelized preprocessing algorithms for high-density oligonucleotide array data

affyPara: Parallelized preprocessing algorithms for high-density oligonucleotide array data Markus Schmidberger Ulrich Mansmann UseR 2008 IBE August 12-14, Technische Universitt Dortmund, http://ibe.web.med.uni-muenchen.de Germany

359 views • 23 slides
Parallelized Kalman-Filter-Based Reconstruction of Particle Tracks with Accurate Detector

Parallelized Kalman-Filter-Based Reconstruction of Particle Tracks with Accurate Detector

Parallelized Kalman-Filter-Based Reconstruction of Particle Tracks with Accurate Detector Geometry G. Cerati 4 , P. Elmer 3 , M. Kortelainen 4 , S. Krutelyov 1 , S. Lantz 2 , M. Lefebvre 3 , M. Masciovecchio 1 , K. McDermott 2 , D. Riley 2 ,

406 views • 26 slides
Parameter efficient training of deep convolutional neural networks by dynamic sparse

Parameter efficient training of deep convolutional neural networks by dynamic sparse

Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization Hesham Mostafa (Intel AI) Xin Wang (Intel AI, Cerebras Systems) Easy : post-training (sparse) compression Hard : direct training of sparse

727 views • 9 slides
Industrial Level Deep Learning Training Infrastructure the Practice and Experience from

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from SenseTime Shengen Yan SenseTime Group Limited. The Success of Deep Learning Google Search AlexNet won ImageNet 2006-01 2007-01 2008-01 2009-01

498 views • 25 slides
Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1 Quick Recap: Training a network Divergence between desired output and Average over all actual output of net for a given input training instances

1.75k views • 140 slides
Accelerate Deep Learning Training at Scale on GPUs Maggie Zhang ( ) maggiez@nvidia.com

Accelerate Deep Learning Training at Scale on GPUs Maggie Zhang ( ) maggiez@nvidia.com

Accelerate Deep Learning Training at Scale on GPUs Maggie Zhang ( ) maggiez@nvidia.com Introduction Why do we need to scale training AGENDA How to achieve scaling DL Training: from single GPU to multi-node 1.33 Minutes on

894 views • 47 slides
Paper number: 230 Training deep segmenta.on networks on texture- encoded input: applica.on to

Paper number: 230 Training deep segmenta.on networks on texture- encoded input: applica.on to

Paper number: 230 Training deep segmenta.on networks on texture- encoded input: applica.on to neuroimaging of the developing neonatal brain AE AE Fe Fe&t, J Cupi., T Kart, D Rueckert The shape hypothesis in deep CNNs cat

476 views • 13 slides
A Parallelized Theorem Prover for Interactive Theorem Proving David L. Rager, Warren A. Hunt Jr.,

A Parallelized Theorem Prover for Interactive Theorem Proving David L. Rager, Warren A. Hunt Jr.,

Introduction Parallelism Primitives Parallelizing ACL2 Evaluate Approach Conclusion A Parallelized Theorem Prover for Interactive Theorem Proving David L. Rager, Warren A. Hunt Jr., and Matt Kaufmann ragerdl@defthm.com, hunt@cs.utexas.edu,

835 views • 59 slides
Deep Vacuum Principles and Application Training Wednesday, February 8, 2012 Introduction Deep

Deep Vacuum Principles and Application Training Wednesday, February 8, 2012 Introduction Deep

Deep Vacuum Principles and Application Training Wednesday, February 8, 2012 Introduction Deep Vacuum Method of evacuation is the only method to use to be sure the system is thoroughly dry and free of non-condensibles and leaks. Wednesday,

646 views • 45 slides
Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2

Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2

Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2 Sami Kiminki Aarno P Antti Immonen 2 Vesa Hirvisalo Jussi Ryyn anen Tommi Zetterman Aalto University Aalto University Nokia Research Center

600 views • 17 slides
No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on Mathematical Theory of Deep Neural Networks October 31, 2019 New York City, NY Need for New Optimization Theory The mystery of deep neural networks

549 views • 17 slides
Training deep Autoencoders for collaborative filtering Oleksii Kuchaiev & Boris Ginsburg

Training deep Autoencoders for collaborative filtering Oleksii Kuchaiev & Boris Ginsburg

Training deep Autoencoders for collaborative filtering Oleksii Kuchaiev & Boris Ginsburg Motivation Personalized recommendations 2 Key points (spoiler alert) 1. Deep autoencoder for collaborative filtering 1. Improves generalization

461 views • 24 slides
The Elliptic Curves Discrete Logarithm Problem and an implementation of parallelized Pollards

The Elliptic Curves Discrete Logarithm Problem and an implementation of parallelized Pollards

What is an elliptic curve? Elliptic Curves in Cryptography. ECDLP resolution. -Pollard and CUDA. Conclusions The Elliptic Curves Discrete Logarithm Problem and an implementation of parallelized Pollards algorithm for ECDLP Alberto

1.52k views • 103 slides
Deep Learning in Pulmonary Image Analysis with Incomplete Training Samples Ziyue Xu, Staff

Deep Learning in Pulmonary Image Analysis with Incomplete Training Samples Ziyue Xu, Staff

Deep Learning in Pulmonary Image Analysis with Incomplete Training Samples Ziyue Xu, Staff Scientist, National Institutes of Health Nov. 2nd, 2017 (GTC DC Talk DC7137) Image Analysis Arguably the most successful application of deep

469 views • 18 slides
Multicore OSes: Looking Forward from 1991, er, 2011 David A. Holland and Margo I. Seltze (Harvard

Multicore OSes: Looking Forward from 1991, er, 2011 David A. Holland and Margo I. Seltze (Harvard

Multicore OSes: Looking Forward from 1991, er, 2011 David A. Holland and Margo I. Seltze (Harvard University) 18.01.2012, Presentation By Pawe Hajdan software is not parallelized enough to take advantage of the more and more parallelized

644 views • 16 slides
A Simple and Easily Parallelized Video Copy Detection Method G. Roth, R. Laganire, M. Bouchard,

A Simple and Easily Parallelized Video Copy Detection Method G. Roth, R. Laganire, M. Bouchard,

A Simple and Easily Parallelized Video Copy Detection Method G. Roth, R. Laganire, M. Bouchard, T. Janati, I . Lakhmirie School of Information Technology and Engineering (SITE) University of Ottawa, Ottawa ON Canada G. Roth 2009 Video Copy

434 views • 25 slides

More recommend