dee deep le p learni arning ng mod models els
play

DEE DEEP LE P LEARNI ARNING NG MOD MODELS ELS Mathew Salvaris - PowerPoint PPT Presentation

Jan 18, 2024 •298 likes •610 views

DIS DISTRI TRIBUT BUTED ED TRA TRAINI INING NG OF OF DEE DEEP LE P LEARNI ARNING NG MOD MODELS ELS Mathew Salvaris @msalvaris Ilia Karmanov @ikdeepl Miguel Fierro @miguelgfierro Rosetta Stone of

  1. DIS DISTRI TRIBUT BUTED ED TRA TRAINI INING NG OF OF DEE DEEP LE P LEARNI ARNING NG MOD MODELS ELS Mathew Salvaris @msalvaris Ilia Karmanov @ikdeepl Miguel Fierro @miguelgfierro

  2. Rosetta Stone of Deep Learning more info: https://github.com/ilkarman/DeepLearningFrameworks Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  3. ImageNet Competition error (%) ImageNet top-5 error 15.3% 7.3% 6.7% 5.1% (human) 3.8% 3.8% 3.6% 3.1% 2.4% Inception- ResNext ResNet NASNet AmoebaNet AlexNet VGG Inception ResNet Instagram (2015) (2017) (2017) (2012) (2014) (2015) (2016) (2018) Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  4. Distributed training mode: Data parallelism Worker 1 Worker 2 Job manager CNN model Subset 1 CNN model Subset 2 CNN model Dataset Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  5. Distributed training mode: Model parallelism Worker 1 Worker 2 Job manager Submodel 1 Submodel 2 CNN model Dataset Dataset Dataset Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  6. Data parallelism vs model parallelism Da Data ta parallelism arallelism Mod odel el parallelism arallelism ▪ Easier implementation ▪ Better scalability of large models ▪ Stronger fault tolerance ▪ Less memory on each GPU ▪ Higher cluster utilization Why no Wh y not bo t both th? ? Da Data ta para aralleli llelism sm fo for r CN CNN lay N layers ers an and model del par aralle allelism lism in in FC FC la laye yers rs so source: ce: Alex ex Krizhevs zhevsky ky. . 2014. On One weird rd trick ck fo for pa paralleli lelizing zing co convolutio lutional nal neura ural l netwo works. rks. https ps://a //arxiv.o rxiv.org/a rg/abs bs/14 /1404.5 04.5997 Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  7. Training strategies: parameter averaging Worker 1 Worker 2 Subset 1 CNN model Subset 2 CNN model Average of weights for each worker Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  8. Training strategies: distributed gradient based Worker 1 Worker 2 Subset 1 CNN model Subset 2 CNN model Synchronous Gradients of each worker Asynchronous Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  9. Overview of distributed training Install software and containers Schedule jobs Provision clusters of VMs Share results Distribute data Scale resources Handling failures Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  10. Azure Distributed Platforms ▪ Batch AI ▪ Batch Shipyard ▪ DL Workspace Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  11. Batch Shipyard • Supports Docker and Singularity: run your Docker and Singularity containers within the same job, side-by-side or even concurrently • Move data easily between locally accessible storage systems, remote filesystems, Azure Blob or File Storage, and compute nodes • Supports local storage, Azure Blob or File Storage, and NFS. • Low priority nodes https://github.com/Azure/batch-shipyard Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  12. Batch AI • Supports running on Docker container as well as the Data Science Virtual Machine • Supports local storage, Azure Blob or File Storage, and NFS. • Low priority nodes https://github.com/Azure/BatchAI Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  13. DL Workspace • Runs jobs inside Docker • Uses Kubernetes • Can be deployed anywhere not just Azure • Supports local storage and NFS https://github.com/Microsoft/DLWorkspace Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  14. Training with Batch AI 1) Create scripts to run on Batch AI 1 and transfer them to file storage I 2) Write the data to storage A I 2 3) Create the docker containers for each DL framework and transfer them to a container registry 3 Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  15. 1) Create a Batch AI Pool I A I 2 2) Each job will pull in the 2 appropriate container, script and 1 Batch AI Pool load data from chosen storage 3) Once the job is completed all the 2 results will be written to the fileshare 3 Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  16. Batch AI Interface CLI Python SDK az batchai cluster create --name nc24r --image UbuntuLTS --vm-size Standard_NC24rs_v3 --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key --nfs $NFS_NAME --nfs-mount-path nfs Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  17. Distributed training with NFS ▪ Batch AI cluster configuration with Copy Data NFS share Batch AI Pool NFS I Share A I az batchai cluster create Mounted --name nc24r Fileshare --image UbuntuLTS --vm-size Standard_NC24rs_v3 --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key --nfs $NFS_NAME --nfs-mount-path nfs Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  18. Distributed training with blob storage ▪ Batch AI cluster configuration with Copy Data mounted blob Batch AI Pool Mounted I Blob A I az batchai cluster create Mounted --name nc24r Fileshare --image UbuntuLTS --vm-size Standard_NC24rs_v3 --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --container-name $CONTAINER_NAME --container-mount-path extcn --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  19. Distributed training with local storage ▪ Batch AI cluster configuration with Copy Data copying the data to the nodes Batch AI Pool I A I az batchai cluster create --name nc24r Mounted --image UbuntuLTS Fileshare --vm-size Standard_NC24r --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --container-name $CONTAINER_NAME --container-mount-path extcn --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key Node preparation configuration -c cluster.json Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  20. Distributed training Results images/second Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  21. Distributed training Results images/second Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  22. Distributed training Results images/second Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  23. Distributed training with Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  24. Distributed training with Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  25. Distributed training with Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  26. Distributed training with PyTorch Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  27. Distributed training with Chainer Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  28. Distributed training with CNTK 1-bit SGD with MPI Blocked Momentum with MPI Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  29. Demo Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  30. Acknowledgements Hongzhi Li Alex Sutton Alex Yukhanov Attribution of some images: http://morguefile.com/ Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  31. Thanks! Mathew Salvaris @msalvaris Ilia Karmanov @ikdeepl Miguel Fierro @miguelgfierro

Recommend

BUI UILDI DING NG A ROB OBUST ON ONLINE LEARNI ARNING NG S STRUCTURE ON RE ON THE OL

BUI UILDI DING NG A ROB OBUST ON ONLINE LEARNI ARNING NG S STRUCTURE ON RE ON THE OL

BUI UILDI DING NG A ROB OBUST ON ONLINE LEARNI ARNING NG S STRUCTURE ON RE ON THE OL OLC P PILLAR ARS OF OF QUAL ALITY BY HAP AZIZ, ED.D. TWITTER: @DIGITALHAP CONTACT: HAPAZIZ@GMAIL.COM BLOG: HTTP://HAPAZIZ.WORDPRESS.COM Session

1.04k views • 17 slides
Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go Going ful g full B Baye yesi sian Bayes = Probabilities Hypothesis = Model Bayes Theorem p ( H | E ) = p ( E | H ) p ( H ) p ( E )

776 views • 61 slides
Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go Going ful g full Bay ayes esian an Bayes = Probabilities Hypothesis = Model Bayes Theorem Evidence = data Prof. Leal-Taix and Prof.

688 views • 66 slides
Excitation Deep Learning Model Abdul Qayyum ImViA laboratory, University of Bourgogne

Excitation Deep Learning Model Abdul Qayyum ImViA laboratory, University of Bourgogne

Medi dical cal Imagin ging g with th Deep ep Learni ning ng 2020 20 Short t Paper er MID IDL L 2020 20 Segmentation of the Myocardium on Late-Gadolinium Enhanced MRI based on 2.5 D Residual Squeeze and Excitation Deep Learning

203 views • 5 slides
Wi Winte ter r Prof ofessi essional onal Learni arning ng Conf nfer erence ence Febr

Wi Winte ter r Prof ofessi essional onal Learni arning ng Conf nfer erence ence Febr

Richard Woods, Georgias School Superintendent Educating Georgias Future gadoe.org Career eer, , Tec echni hnical cal and nd Agricu ricult ltur ure e Educ ucati tion on Wi Winte ter r Prof ofessi essional onal

595 views • 40 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
Motivation Issues of the current live based educational system (DVTS or Vidyo etc) to the

Motivation Issues of the current live based educational system (DVTS or Vidyo etc) to the

Nopphol Pausawasdi, Mahidol University Siriraj Hospital On On demand demand 4K 4K Telemedici dicine ne E learni arning ng Syste System in in Cloud Cloud En Envir vironm nment Nopphol Pausawasdi (Siriraj Hospital, Mahidol Univ. ,

409 views • 8 slides
ICES S TAND ARD - BAS ED S ERVICES NDARD ASED Mariane COMTE University of Montpellier r Event

ICES S TAND ARD - BAS ED S ERVICES NDARD ASED Mariane COMTE University of Montpellier r Event

.eu http:/ p://ww www.l .learni arning ng-com compa pass.e C OM ING L IFEL ONG L EARNIN RNING OMPOS OSIN ELON O PPOR NITY P ATHW HWAYS THRO GH ORTU TUNIT ROUGH ICES S TAND ARD - BAS ED S ERVICES NDARD ASED Mariane COMTE University

1.38k views • 69 slides
Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting Wang* 2 , Quanshi Zhang 1 , Runjin Chen 1 , Di He 2 , Xing Xie 2 * Equal Contribution 1 John Hopcroft Center and the MoE Key Lab of Artificial

219 views • 11 slides
RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab

RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab

RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab luzhc@dgp.toronto.edu 1 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng and Christopher Potts. Recursive Deep Models for Semantic

518 views • 29 slides
BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For Group Activity Recognition Deng et al. BMVC 2015 Simon Fraser University, SPORTLOGIQ , Canada Overview 25 April 2016 Deep Structured Models For

701 views • 40 slides
Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant recognition in the ventral stream Center Name Presenter Name Towards a theory of the above tomaso poggio, CBMM, BCS, CSAIL, McGovern MIT Plan

1.14k views • 81 slides
Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-10-04 Roadmap Challenges of Unstructured Modeling Using Graphs to Describe Model Structure

974 views • 34 slides
Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

A New Framework For Hybrid Models An Application: Deep Hybrid Models Supervised and Semi-Supervised Experiments Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and Stefano Ermon Department of Computer

523 views • 29 slides
Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models have shown promising results in many domains. However, such models often have millions of parameters. The deep learning models face the following

314 views • 16 slides
Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye

Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye

Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan *equal contribution Predictive Models Hybrid Models with Deep and Invertible Features Predictive Models

809 views • 16 slides
Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation Outline Exploring Context with Deep Structured Models Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel; Efficient Piecewise Training

992 views • 45 slides
Earl y w arning s y stems D E FE N SIVE R P R OG R AMMIN G Dr . Colin Gillespie J u mping Ri v

Earl y w arning s y stems D E FE N SIVE R P R OG R AMMIN G Dr . Colin Gillespie J u mping Ri v

Earl y w arning s y stems D E FE N SIVE R P R OG R AMMIN G Dr . Colin Gillespie J u mping Ri v ers Earl y w arning s y stems A v oid problems w here possible Handle iss u es as the y arise in a sensible w a y As an e x ample , u sing the shortc u

661 views • 26 slides
Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

1.44k views • 86 slides
Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

757 views • 22 slides
LEARNI NG TH RO UG H CULTURE Cul t ur al Coope r a t i on a nd I ns pi r a t i on TRANS

LEARNI NG TH RO UG H CULTURE Cul t ur al Coope r a t i on a nd I ns pi r a t i on TRANS

LEARNI NG TH RO UG H CULTURE Cul t ur al Coope r a t i on a nd I ns pi r a t i on TRANS EURO PE H ALLES Pr e s e nt e d by M oukht a r Koc a c he

720 views • 30 slides
Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu <

Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu <

Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu < caseychu@stanford.edu > Jose Blanchet Peter Glynn Deep generative models Deep generative models Variational inference Deep generative models

275 views • 16 slides
Deep Gaussian Mixture Models Cinzia Viroli (University of Bologna, Italy) joint with Geoff

Deep Gaussian Mixture Models Cinzia Viroli (University of Bologna, Italy) joint with Geoff

Deep Gaussian Mixture Models Cinzia Viroli (University of Bologna, Italy) joint with Geoff McLachlan (University of Queensland, Australia) JOCLAD 2018, Lisbona, April 5th, 2018 Outline Deep Learning Mixture Models Deep Gaussian Mixture

915 views • 66 slides
Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Universit Paris Fabien.Moutarde@mines-paristech.fr

777 views • 23 slides

More recommend