node level deep learning input
play

Node-Level Deep Learning Input Pipeline Optimization on - PowerPoint PPT Presentation

Jan 03, 2024 •261 likes •408 views

Node-Level Deep Learning Input Pipeline Optimization on GPGPU-Accelerated HPC Systems 28 Mar 2018 Captain Justin Fletcher Air Force Research Laboratory Integrity Service Excellence Distribution A. Approved for public release;

  1. Node-Level Deep Learning Input Pipeline Optimization on GPGPU-Accelerated HPC Systems 28 Mar 2018 Captain Justin Fletcher Air Force Research Laboratory Integrity  Service  Excellence Distribution A. Approved for public release; distribution unlimited.

  2. Introduction  Deep Learning on high performance computing (HPC) systems has unique challenges  Shared (contested) I/O bandwidth  Distributed file systems  High compute density  From this talk you should get:  The relationship between DL concurrency and I/O  A simple, effective method for hiding I/O on existing systems  An appreciation for the importance of specialized I/O systems for DL on HPC Distribution A. Approved for public release; distribution unlimited.

  3. Motivation  Deep learning on modern HPC systems is bound by system-wide I/O  TensorFlow optimization step running time on 4x P100, 20-core Power8 node is ~50ms/step, when:  System under full, mostly DL, load  LeNet-5 training on MNIST (batch size 256)  Standard asynchronous data queue  Typically ~16 jobs running on separate nodes  Loading the dataset into memory yields ~17x speedup (~3ms)  Instrumenting the training shows exhausted input HPC is likely to remain I/O bound for the foreseeable future [1] queues Distribution A. Approved for public release; distribution unlimited.

  4. Impact of Queue Exhaustion Queue Exhausted → Step Running Time Increases Distribution A. Approved for public release; distribution unlimited.

  5. Causes of Queue Exhaustion: Data Parallelism TF Queue Enqueue Thread 1 Model Copy 1 NIC Element 1 1 Enqueue Thread 2 Element 2 Model Copy 2 Off NIC Enqueue Thread 3 Element 3 2 Node Dequeue … Storage Model Copy 3 Batch Enqueue Thread 4 … Element Q Data Q … NIC Model Copy 4 Enqueue Thread N M  Dequeue rate exceeds enqueue rate  Data parallel concurrency scheme increases dequeue rate  Enqueue threads share storage I/O bandwidth  More model copies = more data throughput  Exacerbated by large data elements Distribution A. Approved for public release; distribution unlimited.

  6. Causes of Queue Exhaustion: Pipeline Parallelism TF Queue Enqueue Thread 1 Model Ops 1 NIC Element 1 1 Enqueue Thread 2 Element 2 Model Ops 2 Off NIC Enqueue Thread 3 Element 3 2 Node Dequeue … Storage Model Ops 3 Batch Enqueue Thread 4 … Element Q Data Q … NIC Model Ops 4 Enqueue Thread N M  Dequeue rate exceeds enqueue rate  Model or pipeline parallel schemes increase dequeue rate  Typically used when model won’t fit on one device  Pipelines operations, increasing throughput Distribution A. Approved for public release; distribution unlimited.

  7. Standard Approach: Increase Thread Count  Enqueue threads asynchronously enqueue data element  Adding more enqueue threads:  Delays queue exhaustion  Decreases slowdown cause by exhaustion  We need to increase the net enqueue rate further  We can’t increase enqueue rate…  So, we must decease the dequeue rate. Distribution A. Approved for public release; distribution unlimited.

  8. Batch Repetition  Artificially slow the dequeue rate by dequeuing batches less than once per step  Allows the queue to fill up  Trivial to implement  Repeating batches introduces new problems  The model is optimized with less new data/step  Your epochs per second will decrease  Generalization is more impacted by how representative any individual batch is of the true data distribution Distribution A. Approved for public release; distribution unlimited.

  9. Batch Repetition Prevents Queue Exhaustion Distribution A. Approved for public release; distribution unlimited.

  10. Batch Interval Impact on Net Enqueue Rate  Running time is inversely proportional to net enqueue rate  Validates the hypothesis that training was I/O bound  We get diminishing returns for batch intervals >16  Batch intervals allow for better throughput with less threads Distribution A. Approved for public release; distribution unlimited.

  11. Batch Interval & Model Performance Distribution A. Approved for public release; distribution unlimited.

  12. Summary  HPC systems are structurally likely to be I/O bound for DL workloads  Repeating batches for an interval of steps can hide I/O latency and keep the GPUs fed  Small refresh intervals don’t impact converged optimization, but decrease runtime  If you want to talk more, ask me about my circular data queues Distribution A. Approved for public release; distribution unlimited.

  13. Distribution A. Approved for public release; distribution unlimited.

  14. HPCMP Distro A Source Distribution A. Approved for public release; distribution unlimited.

Recommend

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides
Andrej Karpathy Bay Area Deep Learning School, 2016 So far... So far... Some input vector (very

Andrej Karpathy Bay Area Deep Learning School, 2016 So far... So far... Some input vector (very

Andrej Karpathy Bay Area Deep Learning School, 2016 So far... So far... Some input vector (very few assumptions made). In many real-world applications input vectors have structure . Spectrograms Text Images Convolutional Neural Networks: A

1.27k views • 104 slides
Industrial Level Deep Learning Training Infrastructure the Practice and Experience from

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from SenseTime Shengen Yan SenseTime Group Limited. The Success of Deep Learning Google Search AlexNet won ImageNet 2006-01 2007-01 2008-01 2009-01

498 views • 25 slides
TOWARDS CREATING A KNOWLEDGE GAP FOR DEEP LEARNING BASED MEDICAL IMAGE ANALYSIS Dr. S.

TOWARDS CREATING A KNOWLEDGE GAP FOR DEEP LEARNING BASED MEDICAL IMAGE ANALYSIS Dr. S.

TOWARDS CREATING A KNOWLEDGE GAP FOR DEEP LEARNING BASED MEDICAL IMAGE ANALYSIS Dr. S. Kevin Zhou, Chinese Academy of Sciences Deep learning Input Algorithm: Output image variable Deep network X Y Y = f(X; W) Learning: arg min W S

685 views • 36 slides
CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view K-NN Linear Classifier Zsolt Kira Georgia Tech Last Time High-level intro to what deep learning is Fast brief of logistics

1.24k views • 100 slides
Session 14 Demystifying Neural Networks Overview The model: An input node for every

Session 14 Demystifying Neural Networks Overview The model: An input node for every

Session 14 Demystifying Neural Networks Overview The model: An input node for every determining variable A specified number of internal nodes An output node for each component of the result Form linear functions of the

672 views • 14 slides
Can Increasing Input Dimensionality Improve Deep Reinforcement Learning? Kei Ota 1 , Tomoaki Oiki

Can Increasing Input Dimensionality Improve Deep Reinforcement Learning? Kei Ota 1 , Tomoaki Oiki

Can Increasing Input Dimensionality Improve Deep Reinforcement Learning? Kei Ota 1 , Tomoaki Oiki 1 , Devesh K. Jha 2 , Toshisada Mariyama 1 , and Daniel Nikovski 2 1. Mitsubishi Electrics, Kanagawa, JP 2. Mitsubishi Electric Research Labs, MA,

305 views • 16 slides
EE-559 Deep learning 9.3. Visualizing the processing in the input Fran cois Fleuret

EE-559 Deep learning 9.3. Visualizing the processing in the input Fran cois Fleuret

EE-559 Deep learning 9.3. Visualizing the processing in the input Fran cois Fleuret https://fleuret.org/ee559/ Aug 17, 2020 Occlusion sensitivity Fran cois Fleuret EE-559 Deep learning / 9.3. Visualizing the processing in the

373 views • 33 slides
STARTING A DEEP LEARNING PROJECT Bryan Catanzaro, 11 May 2017 Supervised learning (learning

STARTING A DEEP LEARNING PROJECT Bryan Catanzaro, 11 May 2017 Supervised learning (learning

STARTING A DEEP LEARNING PROJECT Bryan Catanzaro, 11 May 2017 Supervised learning (learning from tagged data) X Y Input Output tag: Yes/No Image (Is it a coffee mug?) Yes Data: No Learning X Y mappings is hugely useful Andrew Ng

347 views • 23 slides
Anonymity and Censorship Resistance Entry node Middle node Exit node Tor user Tor Node Tor

Anonymity and Censorship Resistance Entry node Middle node Exit node Tor user Tor Node Tor

Anonymity and Censorship Resistance Entry node Middle node Exit node Tor user Tor Node Tor Node Tor Node Tor Node Tor Node Tor Node Tor Node Tor Node Tor Node Tor Node Tor Node Tor Network Encrypted tunnel Web server Unencrypted

575 views • 36 slides
Lecture 5 Lecture 5 Previously in Opt 2 The maximum flow problem Input G = (N, E),

Lecture 5 Lecture 5 Previously in Opt 2 The maximum flow problem Input G = (N, E),

Lecture 5 Lecture 5 Previously in Opt 2 The maximum flow problem Input G = (N, E), a directed graph The node set N contains: A source node, s A sink node, t u ij = edge capacity; u ij 0 Objective: To maximize

789 views • 74 slides
Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
Deep learning 10.2. Causal convolutions Fran cois Fleuret https://fleuret.org/dlc/ Dec 20,

Deep learning 10.2. Causal convolutions Fran cois Fleuret https://fleuret.org/dlc/ Dec 20,

Deep learning 10.2. Causal convolutions Fran cois Fleuret https://fleuret.org/dlc/ Dec 20, 2020 If we use an autoregressive model with a masked input f : { 0 , 1 } T R T R C the input differs from a position to another. During

711 views • 34 slides
A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief

A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief

CS 486/686 Lecture 21 A brief history of deep learning model of neurons in our brain. The output depends only on inputs. Model this by multiplying each input by a continuous valued weight. strengths. Links between neurons are called

392 views • 13 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

307 views • 26 slides
DeepType: On-Device Deep Learning for Input Personalization Service with Minimal Privacy Concern

DeepType: On-Device Deep Learning for Input Personalization Service with Minimal Privacy Concern

DeepType: On-Device Deep Learning for Input Personalization Service with Minimal Privacy Concern Mengwei Xu 1 , Feng Qian 2 , Qiaozhu Mei 3 Kang Huang 4 , Xuanzhe Liu 1 , Yun Ma 1 1 Peking University, 2 University of Minnesota, 3 University of

293 views • 26 slides
Intelligent Low-level Signal Detection and Zero- Suppession in Raw LArTPC Waveforms through Deep

Intelligent Low-level Signal Detection and Zero- Suppession in Raw LArTPC Waveforms through Deep

Intelligent Low-level Signal Detection and Zero- Suppession in Raw LArTPC Waveforms through Deep Learning Techniques Lorenzo Uboldi Supervisor Michael Wang Final report 24 September 2019 DUNE Deep Underground Neutrino Experiment is a LArTPC

492 views • 19 slides
NCI-DOE Cancer Initiative: Ras Biology in Membranes Molecular level Deep Learning (Towards

NCI-DOE Cancer Initiative: Ras Biology in Membranes Molecular level Deep Learning (Towards

NCI-DOE Cancer Initiative: Ras Biology in Membranes Molecular level Deep Learning (Towards Predictive Biology Through HPC) GTC 2017 Brian Van Essen Computer Scientist May 9, 2017 LLNL-PRES-730749 This work was performed under the auspices of

532 views • 32 slides
Deep learning 10.2. Causal convolutions Fran cois Fleuret https://fleuret.org/ee559/ Nov 1,

Deep learning 10.2. Causal convolutions Fran cois Fleuret https://fleuret.org/ee559/ Nov 1,

Deep learning 10.2. Causal convolutions Fran cois Fleuret https://fleuret.org/ee559/ Nov 1, 2020 If we use an autoregressive model with a masked input f : { 0 , 1 } T R T R C the input differs from a position to another. During

772 views • 34 slides
C-Brain: A Deep Learning Accelerator that Tames the Diversity of CNNs through Adaptive Data-level

C-Brain: A Deep Learning Accelerator that Tames the Diversity of CNNs through Adaptive Data-level

C-Brain: A Deep Learning Accelerator that Tames the Diversity of CNNs through Adaptive Data-level Parallelization Lili Song, Ying Wang, Yinhe Han, Xin Zhao, Bosheng Liu, Xiaowei Li State Key Laboratory of Computer Architecture Institute of

552 views • 27 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine learning pipeline Temporal Frame-level Unsupervised Classifier Summary

613 views • 27 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides

More recommend