end to end optimization stack for deep learning
play

End to End Optimization Stack for Deep Learning Presenter: Tianqi - PowerPoint PPT Presentation

Apr 17, 2023 •341 likes •822 views

End to End Optimization Stack for Deep Learning Presenter: Tianqi Chen Paul G. Allen School of Computer Science & Engineering University of Washington Collaborators University of Washington AWS AI Team Ziheng Jiang Tianqi Chen Thierry

  1. End to End Optimization Stack for Deep Learning Presenter: Tianqi Chen Paul G. Allen School of Computer Science & Engineering University of Washington

  2. Collaborators University of Washington AWS AI Team Ziheng Jiang Tianqi Chen Thierry Moreau Haichen Shen ARM, NNVM pipeline ML, Software Stack Hardware Stack GPU and many more contributors in the DMLC community Carlos Guestrin Luis Ceze Arvind Krishnamurthy

  3. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  4. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  5. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  6. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware Built a new accelerator

  7. Deep Learning System Research is Exciting but Hard Need entire software stack CNTK Frameworks on top of it! 
 Layout transformation Quantization Computational graph Operator kernel optimization Benchmarking …. Operator Libraries cuDNN, NNPack, MKL-DNN Hardware Built a new accelerator

  8. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  9. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  10. Deep Learning System Research is Exciting but Hard CNTK Frameworks Data Layout Optimization Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  11. Deep Learning System Research is Exciting but Hard CNTK Frameworks Data Layout Optimization Operator Fusion Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  12. Deep Learning System Research is Exciting but Hard CNTK Frameworks Data Layout Optimization Operator Fusion Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Need optimized hardware kernel Hardware for each variant, on each hardware! 


  13. Deep Learning System Research is Exciting but Hard CNTK Frameworks Data Layout Optimization Operator Fusion Computational graph Serving Operator Libraries cuDNN, NNPack, MKL-DNN Need optimized hardware kernel Hardware for each variant, on each hardware! 


  14. The End to End System Challenge Frameworks Hardware Back-Ends

  15. The End to End System Challenge Frameworks Hardware Back-Ends

  16. The End to End System Challenge Frameworks Hardware Back-Ends

  17. The End to End System Challenge Frameworks Hardware Back-Ends

  18. The End to End System Challenge Frameworks Hardware Back-Ends

  19. The End to End System Challenge Frameworks Hardware Back-Ends

  20. The End to End System Challenge Frameworks Hardware Back-Ends

  21. The End to End System Challenge Frameworks Intermediate representation Hardware Back-Ends

  22. Computational Graph IR and Remaining Gap Examples: NGraph, XLA, NNVM, DLVM … Computational Graph Auto Differentiation Memory Plan Operator Fusion Backends

  23. Computational Graph IR and Remaining Gap Computational Graph Auto Differentiation Memory Plan Operator Fusion Backends

  24. Computational Graph IR and Remaining Gap Computational Graph Auto Differentiation Memory Plan Operator Fusion too many possible choices: precision, layout, fused pattern, device, threading … Need a low level IR to express them explicitly Backends

  25. TVM: Low Level IR Framework • Concise and compact description NNVM Graph Auto Differentiation • Explicit control on codegen Memory Plan • Ease of deployment TVM • Support new hardware backends Hardware backends

  26. Tensor Index Expression Declaration Compute C = dot(A, B.T) import tvm m, n, h = tvm.var('m'), tvm.var('n'), tvm.var('h') Inputs A = tvm.placeholder((m, h), name='A') B = tvm.placeholder((n, h), name=‘B') k = tvm.reduce_axis((0, h), name=‘k') C = tvm.compute((m, n), lambda i, j: tvm.sum(A[i, k] * B[j, k], axis=k)) Computation Rule Shape of C

  27. Challenge: Hardware Diversities IR

  28. Challenge: Hardware Diversities CPU GPU Accelerators IR

  29. Challenge: Hardware Diversities CPU GPU Accelerators L2 FIFO L3 Unified Memory IR SM SM L2 L2 Buffer subsystem TX/L1 TX/L1 Acc L1D L1I L1D L1I RF RF RF RF implicitly managed mixed explicitly managed

  30. Challenge: Hardware Diversities CPU GPU Accelerators L2 FIFO L3 Unified Memory IR SM SM L2 L2 Buffer subsystem TX/L1 TX/L1 Acc L1D L1I L1D L1I RF RF RF RF implicitly managed mixed explicitly managed Compute primitives scalar vector tensor

  31. Challenge: Hardware Diversities CPU GPU Accelerators L2 FIFO L3 Unified Memory IR SM SM L2 L2 Buffer subsystem TX/L1 TX/L1 Acc L1D L1I L1D L1I RF RF RF RF implicitly managed mixed explicitly managed Compute primitives scalar vector tensor Data type fp32 fp16 int8

  32. Unified Schedule Optimizations for Hardwares Algorithm described in IR Lowering Generated code (LLVM, CUDA, OpenCL…)

  33. Unified Schedule Optimizations for Hardwares Algorithm Scheduling described in IR Optimization Lowering Generated code (LLVM, CUDA, OpenCL…)

  34. Unified Schedule Optimizations for Hardwares Scheduling Optimizations Algorithm Scheduling described in IR Optimization ( ✔ ) Data layout Lowering Generated code (LLVM, CUDA, OpenCL…)

  35. Unified Schedule Optimizations for Hardwares Scheduling Optimizations Algorithm Scheduling described in IR Optimization ( ✔ ) Data layout ( ✔ ) Tiling Lowering Generated code (LLVM, CUDA, OpenCL…)

  36. Unified Schedule Optimizations for Hardwares Scheduling Optimizations Algorithm Scheduling described in IR Optimization ( ✔ ) Data layout ( ✔ ) Tiling Lowering ( ✔ ) Thread cooperation Generated code (LLVM, CUDA, OpenCL…)

  37. Unified Schedule Optimizations for Hardwares Scheduling Optimizations Algorithm Scheduling described in IR Optimization ( ✔ ) Data layout ( ✔ ) Tiling Lowering ( ✔ ) Thread cooperation ( ✔ ) Latency hiding Generated code (LLVM, CUDA, OpenCL…)

  38. Unified Schedule Optimizations for Hardwares Scheduling Optimizations Algorithm Scheduling described in IR Optimization ( ✔ ) Data layout ( ✔ ) Tiling Lowering ( ✔ ) Thread cooperation ( ✔ ) Latency hiding Generated code ( ✔ ) T ensorization (LLVM, CUDA, OpenCL…)

  39. Separation of Compilation and Deployment Compilation Stack TVM Runtimes Framework Frontends Deploy NNVM TVM TVM Graph Module Heavy optimizations Lightweight, 300 to 600 KB

  40. Remote Execution and Profiling Devices with TVM Runtime TVM RPC Server with TVM Compiler

  41. Performance Portable against state of art Raspberry Pi 3 K80, Baseline Baseline: MXNet with OpenBLAS and NNPack MXNet with cuDNN auto tune enabled Two undergrad weeks One grad student month 3000 8 MXNet MXNet NNVM Compiler NNVM Compiler 2250 1.2x 6 Time cost(ms) Time cost(ms) 1500 1.2x 4 750 2.2x 2 11.5x 0 0 ResNet18 MobileNet ResNet18 MobileNet Credit: Leyuan Wang(AWS/UCDavis), Yuwei Hu(TuSimple), Zheng Jiang(AWS/FDU)

  42. Coming Soon: Target New Accelerators Tensorization Latency Hiding FPGA Example for building new hardware backend Open-source soon

  43. NNVM Compiler: Open Compiler for AI Systems Caffe Keras MXNet PyTorch Caffe2 CNTK NNVM CoreML ONNX Graph Optimizations External Support TVM Supported Joint Work with AWS AI Team Work in progress TVM Primitives and DMLC community More hardware Metal OpenCL LLVM CUDA backends X86 AMDGPUs ARM Javascript/WASM

  44. Deep Learning System Research is Exciting but Hard CNTK Frameworks Computational graph Operator Libraries cuDNN, NNPack, MKL-DNN Hardware

  45. Deep Learning System Research is Just Exciting CNTK Frameworks My new optimizations NNVM works on all platforms ! Graph Optimizations TVM I can program my new accelerators from python :) TVM Primitives Hardware

  46. Deep Learning System Research is Just Exciting CNTK Frameworks My new optimizations NNVM works on all platforms ! Graph Optimizations TVM I can program my new accelerators from python :) TVM Primitives Hardware You can be part of it!

Recommend

Optimization for marking and sweeping Optimization for marking Use a marking stack

Optimization for marking and sweeping Optimization for marking Use a marking stack

Optimization for marking and sweeping Optimization for marking Use a marking stack Iterative marking Minimize stack depth to avoid stack overflow Knuth: treat marking stark circularly Kurokawa: remove items from stack that have

612 views • 16 slides
TVM Deep Learning on Bare-Metal Devices Pratyush Patel No OS stack Extend TVM to support

TVM Deep Learning on Bare-Metal Devices Pratyush Patel No OS stack Extend TVM to support

TVM Deep Learning on Bare-Metal Devices Pratyush Patel No OS stack Extend TVM to support bare-metal devices Optimization High-Level Differentiable IR AutoTVM Tensor Expression IR LLVM, CUDA VTA AutoVTA Hardware FPGA ASIC Fleet

1.46k views • 18 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Optimization for Training Deep Models presented by Kan Ren Table of Contents Optimization

Optimization for Training Deep Models presented by Kan Ren Table of Contents Optimization

Optimization for Training Deep Models presented by Kan Ren Table of Contents Optimization for machine learning models Challenges of optimizing neural networks Optimizations algorithms initializations adapting the learning

865 views • 62 slides
An Introduction to Optimization Methods in Deep Learning 1 Yuan YAO HKUST Acknowledgement

An Introduction to Optimization Methods in Deep Learning 1 Yuan YAO HKUST Acknowledgement

An Introduction to Optimization Methods in Deep Learning 1 Yuan YAO HKUST Acknowledgement Feifei Li, Stanford cs231n Ruder, Sebastian (2016). An overview of gradient descent optimization algorithms. arXiv:1609.04747.

1.32k views • 48 slides
Full Stack Deep Learning Troubleshooting Deep Neural Networks Josh Tobin, Sergey Karayev, Pieter

Full Stack Deep Learning Troubleshooting Deep Neural Networks Josh Tobin, Sergey Karayev, Pieter

Full Stack Deep Learning Troubleshooting Deep Neural Networks Josh Tobin, Sergey Karayev, Pieter Abbeel Lifecycle of a ML project Cross-project Per-project infrastructure activities Planning & Team & hiring project setup Data

1.66k views • 142 slides
BAYESIAN GLOBAL OPTIMIZATION Using Optimal Learning to Tune Deep Learning Pipelines Scott Clark

BAYESIAN GLOBAL OPTIMIZATION Using Optimal Learning to Tune Deep Learning Pipelines Scott Clark

BAYESIAN GLOBAL OPTIMIZATION Using Optimal Learning to Tune Deep Learning Pipelines Scott Clark scott@sigopt.com OUTLINE 1. Why is Tuning AI Models Hard? 2. Comparison of Tuning Methods 3. Bayesian Global Optimization 4. Deep Learning

983 views • 67 slides
Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of theoretical studies of deep learning 2. Optimization theory of deep neural networks 1) Gradient finds global optima 2) Gram-Gauss-Newton Algorithm

484 views • 31 slides
Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning

Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning

Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning Summer School Montreal August 9, 2015 Google Proprietary Optimization -Exhaustive search -Random search (genetic algorithms) -Analytical solution

1.1k views • 42 slides
Full Stack Deep Learning Lecture 1: Intro Pieter Abbeel, Sergey Karayev, Josh Tobin Organizer

Full Stack Deep Learning Lecture 1: Intro Pieter Abbeel, Sergey Karayev, Josh Tobin Organizer

Full Stack Deep Learning Lecture 1: Intro Pieter Abbeel, Sergey Karayev, Josh Tobin Organizer Intros Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L1: Intro 1 Guest Instructors Saturday Sunday Sunday

747 views • 32 slides
Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement Learning Quentin Cappart , Emmanuel Goutierre, David Bergman, Louis-Martin Rousseau 1 Research question Bounding mechanisms are critical in the

449 views • 24 slides
Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark

Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark

Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark scott@sigopt.com OUTLINE 1. Why is Tuning Models Hard? 2. Common Tuning Methods 3. Deep Learning Example 4. Tuning Multiple Metrics 5.

615 views • 43 slides
Deep Learning and Mixed Integer Optimization Matteo Fischetti, University of Padova 1 Designing

Deep Learning and Mixed Integer Optimization Matteo Fischetti, University of Padova 1 Designing

Deep Learning and Mixed Integer Optimization Matteo Fischetti, University of Padova 1 Designing and Implementing Algorithms for MINLO, Dagstuhl, February 2018 Machine Learning Example (MIPpers only!): Continuous 0-1 Knapack Problem with a

436 views • 24 slides
Reconciling Traditional Optimization Analyses and Modern Deep Learning: the Intrinsic Learning

Reconciling Traditional Optimization Analyses and Modern Deep Learning: the Intrinsic Learning

Reconciling Traditional Optimization Analyses and Modern Deep Learning: the Intrinsic Learning Rate Kaifeng Lyu * Sanjeev Arora Zhiyuan Li * Tsinghua University Princeton University & IAS Princeton University NeurIPS Dec, 2020 *

603 views • 12 slides
Deep learning Optimization and Regularization in deep networks Hamid Beigy Sharif university of

Deep learning Optimization and Regularization in deep networks Hamid Beigy Sharif university of

Deep learning Deep learning Optimization and Regularization in deep networks Hamid Beigy Sharif university of technology October 9, 2019 Hamid Beigy | Sharif university of technology | October 9, 2019 1 / 57 Deep learning Table of contents 1

661 views • 64 slides
Differentially-Private Deep Learning from an Optimization Perspective Presenter: Liyao Xiang

Differentially-Private Deep Learning from an Optimization Perspective Presenter: Liyao Xiang

Differentially-Private Deep Learning from an Optimization Perspective Presenter: Liyao Xiang Shanghai Jiao Tong University 4/30/2019 Privacy Threat Personal information in big data era Is anonymization sufficient to protect user

463 views • 18 slides
Analyzing Optimization in Deep Learning via Trajectories Nadav Cohen Institute for Advanced Study

Analyzing Optimization in Deep Learning via Trajectories Nadav Cohen Institute for Advanced Study

Analyzing Optimization in Deep Learning via Trajectories Nadav Cohen Institute for Advanced Study Institute for Computational and Experimental Research in Mathematics (ICERM) Workshop on Theory and Practice in Machine Learning and Computer

1.46k views • 115 slides
Deep Learning-Driven Simultaneous Layout Decomposition and Mask Optimization Wei Zhong 1,2 ,

Deep Learning-Driven Simultaneous Layout Decomposition and Mask Optimization Wei Zhong 1,2 ,

Deep Learning-Driven Simultaneous Layout Decomposition and Mask Optimization Wei Zhong 1,2 , Shuxiang Hu 1,2 , Yuzhe Ma 3 , Haoyu Yang 3 , Xiuyuan Ma 1,2 , Bei Yu 3 1 Information Science & Engineering, Dalian University of Technology 2 Key Lab

301 views • 28 slides
CS 4803 / 7643: Deep Learning Topics: Regularization Neural Networks Optimization

CS 4803 / 7643: Deep Learning Topics: Regularization Neural Networks Optimization

CS 4803 / 7643: Deep Learning Topics: Regularization Neural Networks Optimization Computing Gradients Zsolt Kira Georgia Tech Administrativia HW0 Reminder Due: 01/18, 11:55pm Plagiarism No Tolerance

988 views • 84 slides
Optimization of Deep Learning Applications in Highly Heterogeneous Distributed System

Optimization of Deep Learning Applications in Highly Heterogeneous Distributed System

Optimization of Deep Learning Applications in Highly Heterogeneous Distributed System 515030910586 F1503024 Last Edited: May. 17, 2018 Report Date: Apr. 20, 2018 1 Catalog Introduction Related Works Proposed

1.03k views • 48 slides
Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from

Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from

Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Backpropagation learning of a network The algorithm 1. Compute a forward pass on the compute graph (DAG) from the input to all the

407 views • 16 slides
Deep Residual Learning for Portfolio Optimization: With Attention and Switching Modules Jeff

Deep Residual Learning for Portfolio Optimization: With Attention and Switching Modules Jeff

Deep Residual Learning for Portfolio Optimization: With Attention and Switching Modules Jeff Wang, Ph.D. Prepared for NYU FRE Seminar. March 7th, 2019 1 / 48 Overview Study model driven portfolio management strategies Construct

694 views • 48 slides
Node-Level Deep Learning Input Pipeline Optimization on GPGPU-Accelerated HPC Systems 28 Mar

Node-Level Deep Learning Input Pipeline Optimization on GPGPU-Accelerated HPC Systems 28 Mar

Node-Level Deep Learning Input Pipeline Optimization on GPGPU-Accelerated HPC Systems 28 Mar 2018 Captain Justin Fletcher Air Force Research Laboratory Integrity Service Excellence Distribution A. Approved for public release;

408 views • 14 slides
Police Patrol Optimization With Geospatial Deep Reinforcement Learning Presenter: Daniel Wilson

Police Patrol Optimization With Geospatial Deep Reinforcement Learning Presenter: Daniel Wilson

Police Patrol Optimization With Geospatial Deep Reinforcement Learning Presenter: Daniel Wilson Other Contributors: Orhun Aydin Omar Maher Mansour Raad Before we begin I am not a criminologist! We are working with a police department to

711 views • 37 slides

More recommend