less is more accelerating deep neural networks with micro
play

Less is More: Accelerating Deep Neural Networks with Micro-Batching - PowerPoint PPT Presentation

May 10, 2023 •289 likes •551 views

Less is More: Accelerating Deep Neural Networks with Micro-Batching 1 Yosuke Oyama 1, a , Tal Ben-Nun 2 , Torsten Hoefler 2 , Satoshi Matsuoka 1 1) Tokyo Institute of Technology, 2) ETH Zurich a) oyama.y.aa@m.titech.ac.jp, Presenter 162

  1. Less is More: Accelerating Deep Neural Networks with Micro-Batching 1 Yosuke Oyama 1, a , Tal Ben-Nun 2 , Torsten Hoefler 2 , Satoshi Matsuoka 1 1) Tokyo Institute of Technology, 2) ETH Zurich a) oyama.y.aa@m.titech.ac.jp, Presenter � 162 ������������������������ 2017/12/19

  2. Background: cuDNN Convolution 2 ´ NVIDIA cuDNN : A deep learning kernel library for NVIDIA GPUs ´ Adopted by most of deep learning frameworks ´ Contains multiple convolution algorithms for CNNs ´ GEMM, direct, FFT, Winograd, … ´ Most algorithms use workspace : A buffer in GPU memory to store intermediate data N Y [ n , k , h , w ] = X Y Σ c , u , v W [ k , c , u , v ] * X [ n , c , h + u , w + v ]; K C C 1 Σ H W 1 H V 1 U W W 2D Convolution (forward)

  3. Background: cuDNN Convolution 3 ´ Concern: There are considerable performance gaps (w.r.t. time and workspace size) among convolution algorithms ´ e.g. Inappropriate workspace limit on AlexNet leads ~4.51x slowdown 2 20 Execution time [ms] 0 15 0 IMPLICIT_GEMM 1 IMPLICIT_PRECOMP_GEMM 1 10 2 GEMM 3 DIRECT 4 FFT 5 5 FFT_TILING ● 6 WINOGRAD 7 5 4 7 WINOGRAD_NONFUSED 0 0 200 400 600 800 1000 Workspace size [MiB] Execution time vs. workspace size of AlexNet conv2 (forward) Mini-batch size of 256, NVIDIA P100-SMX2, cuDNN 7.0

  4. Background: cuDNN Convolution 4 ´ Concern: There are considerable performance gaps (w.r.t. time and workspace size) among convolution algorithms ´ e.g. Inappropriate workspace limit on AlexNet leads ~4.51x slowdown if workspace If workspace limit < 323 MiB limit � 323 MiB 2 20 Execution time [ms] 0 15 0 IMPLICIT_GEMM 1 IMPLICIT_PRECOMP_GEMM 1 10 2 GEMM 3 DIRECT 4 FFT 5 5 FFT_TILING ● 6 WINOGRAD 7 5 4 7 WINOGRAD_NONFUSED 0 0 200 400 600 800 1000 Workspace size [MiB] Execution time vs. workspace size of AlexNet conv2 (forward) Mini-batch size of 256, NVIDIA P100-SMX2, cuDNN 7.0

  5. Background: cuDNN Convolution 5 ´ Concern: There are considerable performance gaps (w.r.t. time and workspace size) among convolution algorithms ´ e.g. Inappropriate workspace limit on AlexNet leads ~4.51x slowdown if workspace If workspace limit < 323 MiB limit � 323 MiB 2 20 Execution time [ms] 0 15 0 IMPLICIT_GEMM 1 IMPLICIT_PRECOMP_GEMM 1 10 2 GEMM 4.51x 3 DIRECT 4 FFT 5 5 FFT_TILING ● 6 WINOGRAD 7 5 4 7 WINOGRAD_NONFUSED 0 0 200 400 600 800 1000 Workspace size [MiB] Execution time vs. workspace size of AlexNet conv2 (forward) Mini-batch size of 256, NVIDIA P100-SMX2, cuDNN 7.0

  6. Background: cuDNN Convolution 6 ´ Observation: Less batch size for More executable performance ´ Faster algorithms can be enabled by dividing the mini-batch 80 Workspace size [MiB] ● ● 300 ● ● Images/time [ms − 1 ] ● ● ● ● ● ● ● ● ● ● ● ● ● ● 60 ● ● ● ● ● ● ● ● 200 ● ● ● ● ● ● ● ● ● ● ● 40 ● ● ● ● ● ● ● ● ● ● ● ● 100 ● ● ● ● ● ● 20 ● ● ● Images/time [ms − 1 ] ● ● ● ● ● ● Workspace size [MiB] ● 0 0 0 50 100 150 200 250 Batch size Computation performance and workspace size of FFT_TILING of AlexNet conv2 (forward)

  7. Background: cuDNN Convolution 7 ´ Observation: Less batch size for More executable performance ´ Faster algorithms can be enabled by dividing the mini-batch 93% of performance with 58% of workspace 80 Workspace size [MiB] ● ● 300 ● ● Images/time [ms − 1 ] ● ● ● ● ● ● ● ● ● ● ● ● ● ● 60 ● ● ● ● ● ● ● ● 200 ● ● ● ● ● ● ● ● ● ● ● 40 ● ● ● ● ● ● ● ● ● ● ● ● 100 ● ● ● ● ● ● 20 ● ● ● Images/time [ms − 1 ] ● ● ● ● ● ● Workspace size [MiB] ● 0 0 0 50 100 150 200 250 Batch size Computation performance and workspace size of FFT_TILING of AlexNet conv2 (forward)

  8. Approach and Contribution 8 ´ Approach: µ-cuDNN, a wrapper library for cuDNN ´ µ-cuDNN divides one mini-batch into more fine-grained batches (“micro-batches”) for cuDNN convolution ´ µ-cuDNN optimizes micro-batch sizes and algorithms using Dynamic Programming and Integer Linear Programming ´ Contribution: ´ µ-cuDNN on NVIDIA Tesla P100-SMX GPU achieves ´ up to 2.33x speedup for single convolution ´ up to 1.63x speedup for convolution of a CNN

  9. Proposal: µ-cuDNN 9 ´ µ-cuDNN : a C++ transparent wrapper for cuDNN ´ is installed by replacing cudnnHandle_t to UcudnnHandle_t in deep learning frameworks ´ e.g. Caffe requires 3 lines of modification ´ overloads some of cuDNN functions ´ It internally divides cudnnConvolution* into multiple convolutions ´ It delegates other functions to cuDNN itself

  10. Proposal: Workspace policies of µ-cuDNN 10 ´ µ-cuDNN supports two different workspace policies ´ Workspace Reuse (WR) ´ Each layer reuses a private workspace ´ Total workspace size is O(#layer) ´ Workspace Division (WD) ´ Each layer divides an unified workspace and use part of it ´ Total workspace size is constant Workspace Reuse (WR) Workspace Division (WD) Total WS size is up to [WS limit/layer] * [#layer] [total WS limit] µ-batch division is optimized by DP DP + ILP WS is managed by DL frameworks µ-cuDNN WS limit is passed by cuDNN interface An environment variable

  11. Proposal: WR using Dynamic Programming 11 ´ Problem: Given a mini-batch size B and the fastest execution time T μ ( b ) for b =1, 2, …, B , compute � � T µ ( B ) T ( B ) = min min b =1 , 2 ,...,B − 1 T ( b ) + T ( B − b ) ´ and get the mini-batch division (“configuration” in this work) Time T (256) conv1 conv1 conv1 conv1 conv1 batch=60 batch=60 batch=60 batch=60 batch=16 T (60) = T μ (60) T (196)

  12. � � � � � � � � � � � � � � � � � Proposal: WR using Dynamic Programming 12 1. for b � B policy ( B) , benchmarks the fastest execution time T μ ( b ) and its micro-configuration c μ ( b ) = ( a , b ) ´ where a is algorithm ID and B is mini-batch size ´ T μ ( b ) and a are obtain by cudnnFindConvolution*Algorithm ´ B all ( B ) = {1,2,…, B }, B powerOfTwo ( B ) = {2 0 ,2 1 ,…, B }, B undivided ( B ) = { B } 2. for b = 1, 2, …, B , computes b 1 , ˆ ˆ = argmin { T µ ( b 1 ) + T ( b 2 ) } b 2 b 1 + b 2 = b T µ ( ˆ b 1 ) + T ( ˆ T ( b ) = b 2 ) { c µ ( ˆ b 1 ) } ∪ c ( ˆ c ( b ) = b 2 ) 3. outputs configuration (a list of micro-configurations) c ( B )

  13. Proposal: WR using Dynamic Programming 13 Time c (256) = {(4, 60), (4, 60), (4, 60), (4, 60), (0, 16)} T (256) conv1 conv1 conv1 conv1 conv1 c μ = (4, 60) c μ = (4, 60) c μ = (4, 60) c μ = (4, 60) c μ = (0, 16) T μ (60) T (196) c μ (60) = (4, 60) c (196) = {(4, 60), (4, 60), (4, 60), (0, 16)} T μ (60) T (136) c μ (60) = (4, 60) c (136) = {(4, 60), (4, 60), (0, 16)}

  14. � � � � � � � Proposal: WD using Integer LP 14 ´ Problem: X X Total execution time min . T = T k ( c ) x k,c k 2 K c 2 C k Total workspace size should X X subject to . M k ( c ) x k,c ≤ M be less than M � k 2 K c 2 C k Exactly one configuration should X x k,c = 1 ( ∀ k ∈ K ) be selected for each kernel c 2 C k x k , c =1 � configuration c is x k,c ∈ { 0 , 1 } ( ∀ k ∈ K, ∀ c ∈ C k ) selected for kernel k ´ M : total workspace size ´ K : A set of convolution kernels ´ C k : A set of configurations for kernel k ´ T k ( c ), M k ( c ) : Execution time and workspace size for config. c

  15. Proposal: WD using Integer LP 15 Total workspace size x k , c = 1 M conv k x 2, v = 1 c � C k x 1, u = 1 conv2 M 2 ( v ) v � C 2 conv1 T 2 ( v ) u � C 1 Time min. T c μ c μ c μ c μ

  16. Proposal: WD using Integer LP 16 ´ Challenge: How to enumerate practical number of configurations (i.e. 0-1 variables) for each kernel � ´ The total number is Ω (|# algo | B ) ´ Solution: Pruning “undesirable” configurations ´ Definition. A configuration c is desirable among a set C � � c ’ � C , T ( c ) < T ( c ’) � M ( c ) < M ( c ’) T ( c ) c is undesirable, because it is slower than c ’ • c requires more memory than c ’ • c’ M ( c )

  17. Proposal: WD using Integer LP 17 1. For each convolution kernel, list up all “desirable” configurations using the DP ´ We apply the pruning D ( C ) = { c � C | � c ’ � C , T ( c ) < T ( c ’) � M ( c ) < M ( c ’)} for each iteration 2. Pass the output (configurations) to the ILP problem 3. Solve the ILP problem ´ µ-cuDNN uses GNU Linear Programming Kit (GLPK) as a LP solver

  18. Evaluation 18 ´ Software: Caffe 1.0, cuDNN 6.0, CUDA 8.0 ´ All CNN tensors are stored in float, NCHW format ´ Workspace size limit is set to 8, 64, 512 MiB ´ GPUs: ´ NVIDIA Tesla P100-SMX2 @ TSUBAME 3.0 ´ 10.6 SP TFlop/s ´ 24 GiB GDDR5 memory, 480 GiB/s bandwidth ´ NVIDIA Tesla K80 @ TSUBAME-KFC/DL ´ 8.73 SP TFlop/s ´ 16 GiB HBM2 memory, 732 GiB/s bandwidth

  19. Evaluation: WR using Dynamic Programming 19 ´ µ-cuDNN achieved 2.33x speedup on forward convolution of AlexNet conv2 6 IMPLICIT_PRECOMP_GEMM FFT_TILING 5 WINOGRAD_NONFUSED Execution time [ms] 4 256 3 32 32 48 32 2 48 32 48 32 1 32 48 32 32 32 32 0 undivided powerOfTwo all cudnnConvolutionForward of AlexNet conv2 on NVIDIA Tesla P100-SMX2 Workspace size of 64 MiB, mini-batch size of 256 Numbers on each rectangles represent micro-batch sizes

Recommend

Accelerating drug discovery with deep neural networks literature review Tobias Sikosek Senior

Accelerating drug discovery with deep neural networks literature review Tobias Sikosek Senior

Accelerating drug discovery with deep neural networks literature review Tobias Sikosek Senior Data Scientist In Silico Unit (Heidelberg) Deep Learning Artificial intelligence Machine learning Deep learning Renewed focus on multi-layer

836 views • 50 slides
Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the Search for Dark Matter with Machine Learning April 10, 2019 Based on Published in ICLR 2018, htups://arxiv.org/abs/1711.00165 Open source

501 views • 33 slides
A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks Yuzhe Ma 1

A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks Yuzhe Ma 1

A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks Yuzhe Ma 1 , Ran Chen 1 , Wei Li 1 , Fanhua Shang 2 , Wenjian Yu 3 , Minsik Cho 4 , Bei Yu 1 1 CUHK, 2 Xidian Univ., 3 Tsinghua Univ. 4 IBM T. J. Watson 1 / 23

542 views • 23 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Accelerating Deep Learning Frameworks with Micro-batches Yosuke Oyama 1 * Tal Ben-Nun 2 Torsten

Accelerating Deep Learning Frameworks with Micro-batches Yosuke Oyama 1 * Tal Ben-Nun 2 Torsten

Accelerating Deep Learning Frameworks with Micro-batches Yosuke Oyama 1 * Tal Ben-Nun 2 Torsten Hoefler 2 Satoshi Matsuoka 3 1 September 13, 2018 1 Tokyo Institute of Technology 2 ETH Zurich 3 RIKEN Center for Computational Science 1/26 *

605 views • 48 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides
Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G. Montavon, W. Samek, K.-R. Mller Part 2: Making Deep Neural Networks Transparent 5 March 2017 Making Deep Neural Nets Transparent DNN transparency

448 views • 44 slides
Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit: Vlad Morariu Training (Deep) Neural Networks Computational graphs Improvements to gradient descent Stochastic gradient descent Momentum

814 views • 21 slides
6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Monday (2/5) 30 minutes Topics: Optimization Basic neural networks Neural Network

619 views • 41 slides
DEEP NEURAL NETWORKS FOR OBJECT DETECTION Sergey Nikolenko Steklov Institute of Mathematics at

DEEP NEURAL NETWORKS FOR OBJECT DETECTION Sergey Nikolenko Steklov Institute of Mathematics at

DEEP NEURAL NETWORKS FOR OBJECT DETECTION Sergey Nikolenko Steklov Institute of Mathematics at St. Petersburg October 10, 2017, Seoul, Korea Outline Birds eye overview of deep learning Convolutional neural networks Chris

743 views • 32 slides
Weight Parameterizations in Deep Neural Networks Sergey Zagoruyko e Paris-Est, Universit

Weight Parameterizations in Deep Neural Networks Sergey Zagoruyko e Paris-Est, Universit

Weight Parameterizations in Deep Neural Networks Weight Parameterizations in Deep Neural Networks Sergey Zagoruyko e Paris-Est, Universit Ecole des Ponts ParisTech December 26, 2017 Weight Parameterizations in Deep Neural Networks

1.05k views • 45 slides
CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Deep Neural Networks Gradient Vanishing Rectified linear units Overfitting

941 views • 25 slides

More recommend