Transparent parallelization of neural network training Cyprien Noel - PowerPoint PPT Presentation

Mar 02, 2024 •233 likes •562 views

Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015 by overthemoon Outline Neural Nets at Flickr Training Fast Parallel Distributed Q&A Tagging Photos Class Probability

Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015 by overthemoon
Outline ▪ Neural Nets at Flickr ▪ Training Fast ▪ Parallel ▪ Distributed ▪ Q&A
Tagging Photos Class Probability Flowers 0.98 Classifiers Outdoors 0.95 Classifiers Classifiers Cat 0.001 Grass 0.6 Any photo on Flickr is classified using computer vision
Auto Tags Feeding Search ▪ ▪ ▪ ▪ ▪ ▪
Tagging the Flickr corpus ▪ Classify millions of new photos per day ▪ Apply new models to billions of photos ▪ Train new models using Caffe
Training new models ▪ Manual experimentation ▪ Hyperparameter search ▪ Limitation is training time → Parallelize Caffe
Goals ▪ “Transparent” ▪ Code Isolation ▪ Existing Models ▪ Globally connected layers ▪ Existing Infrastructure
Outline ▪ Neural Nets at Flickr ▪ Training Fast ▪ Parallel ▪ Distributed ▪ Q&A
GoogLeNet, 2014
Ways to Parallelize ▪ Model ▪ Caffe team enabling this now ▪ Data ▪ Synchronous ▪ Asynchronous
Outline ▪ Neural Nets at Flickr ▪ Training Faster ▪ Parallel ▪ Distributed ▪ Q&A
First Approach: CPU ▪ Hogwild! (2011) ▪ Cores read and write from shared buffer ▪ No synchronization ▪ Data races surprisingly low
MNIST CPU
Hogwild ▪ Plateaus with core counts ▪ Some Potential ▪ On a grid ▪ With model parallelism
But we are at GTC
GPU Cluster ▪ A lot of time spent preparing experiments ▪ Code Deployment ▪ ▪ Data Handling ▪ On the fly datasets for “big data”
Outline ▪ Neural Nets at Flickr ▪ Training Fast ▪ Parallel ▪ Distributed ▪ Q&A
Second Approach: Lots of Boxes
Second Approach: Lots of Boxes ▪ Exchange gradients between nodes ▪ Parameter server setup ▪ Easy: move data fast
GPU memory - PCI - Ethernet
Second Approach: Lots of Boxes ▪ 230MB * 2 * N per batch ▪ TCP/UDP chokes ▪ Machines unreachable ▪ No InfiniBand or RoCE
Second Approach: Lots of Boxes ▪ Modify Caffe: chunk parameters
Packet_mmap Buffer App Kernel App Kernel
MNIST
ImageNet
NVIDIA ▪ Large Machines ▪ 4 or 8 GPUs ▪ Root PCI switches ▪ InfiniBand
Third Approach: CUDA P2P ▪ GPUs on single machine ▪ Data Feeding ▪ Caffe Pipeline ▪ Async Streams
State of Things ▪ Async ~8x but no momentum ▪ Sync ~2x ▪ Combining both, and model parallelism ▪ Working on auto tuning of params (batch, rate) ▪ Different ratios of compute vs. IO
Takeaway ▪ Check Caffe, including Flickr’s contributions ▪ CUDA + Docker = Love ▪ Small SOC servers might be interesting for ML
Thanks! Flickr vision team Flickr backend team Yahoo labs cypof@yahoo-inc.com

Recommend

Transparent Parallelization of Binary Code Benot Pradelle Alain Ketterlin Philippe Clauss

Transparent Parallelization of Binary Code Benot Pradelle Alain Ketterlin Philippe Clauss Universit de Strasbourg INRIA (CAMUS team, Centre Nancy Grand-Est) CNRS (LSIIT, UMR 7005) First International Workshop on Polyhedral Compilation

305 views • 18 slides

CS4811 Neural Network Training Example Consider the following network. It has two inputs (two

CS4811 Neural Network Training Example Consider the following network. It has two inputs (two entries in each input vector), one hidden layer with two neurons and the output layer with a single neuron. Each neuron has a bias input to allow

216 views • 4 slides

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK LABORATORY FOR NEURAL COMPUTING FOR MACHINE LEARNING DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE YORK UNIVERSITY, TORONTO NOVEMBER

211 views • 20 slides

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi,

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geofgrey Hinton Motivation We need tools to understand trained neural networks Neural network training involves interactions between an

588 views • 16 slides

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent, Momentum, weight decay New: (last 5-6 years) Dropout ReLUs Batch Normalization Reminder: Overfitting, in images Classification just right

1.61k views • 132 slides

FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki

FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki Nakahara, Youki Sada, Masayuki Shimoda, Akira Jinguji, Shimpei Sato Tokyo Institute of Technology, JP FPL2019 @Barcelona Challenges in DL Training

369 views • 10 slides

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

1.79k views • 165 slides

On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center

On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center Georgia Tech The Complexity of Learning Neural Networks Deep learning s successes are incredible Do you want to beat 9 dan Goplayers? The

873 views • 55 slides

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides

Artificial Neural Network : Training Debasis Samanta IIT Kharagpur

Artificial Neural Network : Training Debasis Samanta IIT Kharagpur debasis.samanta.iitkgp@gmail.com 06.04.2018 Debasis Samanta (IIT Kharagpur) Soft Computing Applications 06.04.2018 1 / 49 Learning of neural networks: Topics Concept of

608 views • 49 slides

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Alexander Shevchenko Marco Mondelli Neural Network Training From theoretical perspective training of neural networks is di ffi cult

494 views • 28 slides

Networks Luke Schuler Overview What is an Artificial Neural Network? History

Artificial Neural Networks Luke Schuler Overview What is an Artificial Neural Network? History Inception Abandonment Resurgence Model Training Applications What is an Artificial Neural Network? An artificial

616 views • 15 slides

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides

Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo

Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo Muoz 2 , Sebastian Pokutta 3 January 9, 2019 1 IEOR, Columbia University 2 IVADO, Polytechnique Montral 3 ISyE, Georgia Tech 1 ...Im

861 views • 55 slides

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang &

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang & Phil Woodland University of Cambridge 11 November 2013 Conventional Training of CD-DNN-HMMs CD-DNN-HMMs rely on GMM-HMMs in two aspects:

170 views • 13 slides

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen,

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, Jie Tang Real-world Graphs Biol Bi ologi ogical G Graph Question: Soci So cial

598 views • 20 slides

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G. Montavon, W. Samek, K.-R. Mller Part 2: Making Deep Neural Networks Transparent 5 March 2017 Making Deep Neural Nets Transparent DNN transparency

448 views • 44 slides

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Training Neural Nets 1 / 40 Outline 1 The Softmax Simplex 2 Loss and Risk 3 Back-Propagation 4 Stochastic Gradient Descent 5 Regularization 6 Network Depth

675 views • 40 slides

Training Behavior of Sparse Neural Network Topologies Simon Alford, Ryan Robinett, Lauren

Training Behavior of Sparse Neural Network Topologies Simon Alford, Ryan Robinett, Lauren Milechin, Jeremy Kepner Outline Introduction Approach Results Interpretation and Summary Slide - 2 Limiting factors confronting Deep

984 views • 62 slides

Global Optimality in Neural Network Training Benjamin D. Haeffele and Ren Vidal Johns Hopkins

Global Optimality in Neural Network Training Benjamin D. Haeffele and Ren Vidal Johns Hopkins University, Center for Imaging Science. Baltimore, USA Questions in Deep Learning Architecture Design Optimization Generalization Questions in

1.56k views • 91 slides