putting deep learning models in production sahil dua
play

Putting Deep Learning Models in Production Sahil Dua @sahildua2305 - PowerPoint PPT Presentation

Jun 19, 2023 •504 likes •938 views

Putting Deep Learning Models in Production Sahil Dua @sahildua2305 @sahildua2305 Lets imagine! @sahildua2305 @sahildua2305 But ... @sahildua2305 @sahildua2305 whoami Software Developer @ Booking.com Previously - Deep Learning

  1. Putting Deep Learning Models in Production Sahil Dua @sahildua2305 @sahildua2305

  2. Let’s imagine! @sahildua2305 @sahildua2305

  3. But ... @sahildua2305 @sahildua2305

  4. whoami ➔ Software Developer @ Booking.com ➔ Previously - Deep Learning Infrastructure ➔ Open Source Contributor (Git, Pandas, Kinto, go-github, etc.) ➔ Tech Speaker @sahildua2305 @sahildua2305

  5. Agenda ➔ Deep Learning at Booking.com ➔ Life-cycle of a model ➔ Training Models ➔ Serving Predictions @sahildua2305 @sahildua2305

  6. Deep Learning at Booking.com @sahildua2305 @sahildua2305

  7. Scale highlights . 1,500,000 + 1.4 million + room nights active properties booked in 220+ countries every 24 hours @sahildua2305 @sahildua2305

  8. Deep Learning ➔ Image understanding ➔ Translations ➔ Ads bidding ➔ ... @sahildua2305 @sahildua2305

  9. Image Tagging @sahildua2305 @sahildua2305

  10. Image Tagging @sahildua2305 @sahildua2305

  11. Image Tagging Sea view: 6.38 Balcony/Terrace: 4.82 Photo of the whole room: 4.21 Bed: 3.47 Decorative details: 3.15 Seating area: 2.70 @sahildua2305 @sahildua2305

  12. @sahildua2305 @sahildua2305

  13. Image Tagging Using the image tag information in the right context Swimming pool, Breakfast Buffet, etc. @sahildua2305 @sahildua2305

  14. Lifecycle of a model @sahildua2305 @sahildua2305

  15. Lifecycle of a model Data Train Deploy Analysis @sahildua2305 @sahildua2305

  16. Training a Model - on laptop @sahildua2305 @sahildua2305

  17. Training a Model - on laptop @sahildua2305 @sahildua2305

  18. Machine Learning workload ➔ Computationally intensive workload ➔ Often not highly parallelizable algorithms ➔ 10 to 100 GBs of data @sahildua2305 @sahildua2305

  19. Why Kubernetes (k8s)? ➔ Isolation ➔ Elasticity ➔ Flexibility @sahildua2305 @sahildua2305

  20. Why k8s – GPUs? ➔ In alpha since 1.3 ➔ Speed up 20X-50X resources: limits: alpha.kubernetes.io/nvidia-gpu: 1 @sahildua2305 @sahildua2305

  21. Training with k8s ➔ Base images with ML frameworks ◆ TensorFlow, Torch, VowpalWabbit, etc. ➔ Training code is installed at start time ➔ Data access - Hadoop (or PVs) @sahildua2305 @sahildua2305

  22. Startup Training pod Code .. start.sh train.py evaluate.py @sahildua2305 @sahildua2305

  23. Startup Training pod Data .. start.sh PV train.py evaluate.py @sahildua2305 @sahildua2305

  24. Streaming logs back Training pod Logs .. start.sh PV train.py evaluate.py @sahildua2305 @sahildua2305

  25. Exports the model Training pod model .. start.sh PV train.py evaluate.py @sahildua2305 @sahildua2305

  26. Serving predictions @sahildua2305 @sahildua2305

  27. Serving Predictions Input Features Client Model Prediction @sahildua2305 @sahildua2305

  28. Serving Predictions Input Features Client Model 1 Prediction Input Features Client Model X Prediction @sahildua2305 @sahildua2305

  29. Serving Predictions Input Features Client Model 1 Prediction Input Features Client Model X Prediction @sahildua2305 @sahildua2305

  30. Serving Predictions ➔ Stateless app with common code ➔ Containerized ➔ No model in image ➔ REST API for predictions @sahildua2305 @sahildua2305

  31. Serving Predictions Input App Features Client model Prediction @sahildua2305 @sahildua2305

  32. Serving Predictions ➔ Get trained model from Hadoop ➔ Load model in memory ➔ Warm it up ➔ Expose HTTP API ➔ Respond to the probes @sahildua2305 @sahildua2305

  33. Serving Predictions Input Features Client Prediction @sahildua2305 @sahildua2305

  34. Serving Predictions Input Features Client Prediction Input Features Client Prediction @sahildua2305 @sahildua2305

  35. Deploying a new model ➔ Create new Deployment ➔ Create new HTTP Route ➔ Wait for liveness/readiness probe @sahildua2305 @sahildua2305

  36. Performance PredictionTime = RequestOverhead + N*ComputationTime N is the number of instances to predict on @sahildua2305 @sahildua2305

  37. Optimizing for Latency ➔ Do not predict if you can precompute ➔ Reduce Request Overhead ➔ Predict for one instance ➔ Quantization (float 32 => fixed 8) ➔ TensorFlow specific: freeze network & optimize for inference @sahildua2305 @sahildua2305

  38. Optimizing for Throughput ➔ Do not predict if you can precompute ➔ Batch requests ➔ Parallelize requests @sahildua2305 @sahildua2305

  39. Summary ➔ Training models in pods ➔ Serving models ➔ Optimizing serving for latency/throughput @sahildua2305 @sahildua2305

  40. Next steps ➔ Tooling to control hundred deployments ➔ Autoscale prediction service ➔ Hyper parameter tuning for training @sahildua2305 @sahildua2305

  41. Want to get in touch? LinkedIn / Twitter / GitHub @sahildua2305 Website www.sahildua.com @sahildua2305 @sahildua2305

  42. THANK YOU @sahildua2305 @sahildua2305 @sahildua2305

Recommend

and Features in Deep Learning Interpretation Sahil Singla Joint work with Eric Wallace, Shi

and Features in Deep Learning Interpretation Sahil Singla Joint work with Eric Wallace, Shi

Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation Sahil Singla Joint work with Eric Wallace, Shi Feng, Soheil Feizi University of Maryland Pacific Ballroom # 69 , 6:30-9:00 PM June 13th 2019

650 views • 37 slides
Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-10-04 Roadmap Challenges of Unstructured Modeling Using Graphs to Describe Model Structure

974 views • 34 slides
DEEP LEARNING DEPLOYMENT WITH NVIDIA TENSORRT Shashank Prasanna Deep Learning in Production -

DEEP LEARNING DEPLOYMENT WITH NVIDIA TENSORRT Shashank Prasanna Deep Learning in Production -

DEEP LEARNING DEPLOYMENT WITH NVIDIA TENSORRT Shashank Prasanna Deep Learning in Production - Current Approaches - Deployment Challenges NVIDIA TensorRT - Programmable Inference Accelerator - Performance, Optimizations and Features

438 views • 28 slides
Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models have shown promising results in many domains. However, such models often have millions of parameters. The deep learning models face the following

314 views • 16 slides
Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant recognition in the ventral stream Center Name Presenter Name Towards a theory of the above tomaso poggio, CBMM, BCS, CSAIL, McGovern MIT Plan

1.14k views • 81 slides
Approach to practical application of Deep Learning in manufacturer's production line Masahiro

Approach to practical application of Deep Learning in manufacturer's production line Masahiro

Approach to practical application of Deep Learning in manufacturer's production line Masahiro Kashiwagi, Hiroyuki Kusaka, Kiminori Kurosawa and Kenji Nishide 1 1 Applications Service Agriculture Advertisement Education Security Deep

504 views • 32 slides
models to understand visual cortex 11-785 Introduction to Deep Learning Fall 2017 Michael Tarr

models to understand visual cortex 11-785 Introduction to Deep Learning Fall 2017 Michael Tarr

using deep learning models to understand visual cortex 11-785 Introduction to Deep Learning Fall 2017 Michael Tarr Department of Psychology Center for the Neural Basis of Cognition this lecture A bit out of order Oct 2: Models of

462 views • 42 slides
Evaluation of Deep Learning Evaluation of Deep Learning Models for Network Models for Network

Evaluation of Deep Learning Evaluation of Deep Learning Models for Network Models for Network

Evaluation of Deep Learning Evaluation of Deep Learning Models for Network Models for Network Performance Prediction for Performance Prediction for Scientific Facilities Scientific Facilities Makiya Nakashima: Texas A&M

438 views • 16 slides
Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation Outline Exploring Context with Deep Structured Models Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel; Efficient Piecewise Training

992 views • 45 slides
Deep Learning and Hardware: Matching the Demands from the Machine Learning Community Ekapol

Deep Learning and Hardware: Matching the Demands from the Machine Learning Community Ekapol

Deep Learning and Hardware: Matching the Demands from the Machine Learning Community Ekapol Chuangsuwanich Department of Computer Engineering, Chulalongkorn University Deep learning Artificial Neural Networks rebranded Deeper models Bigger

808 views • 44 slides
Semi-supervised Learning with Deep Generative Models Diedrik P. Kingma, Danilo J. Rezende, Shakir

Semi-supervised Learning with Deep Generative Models Diedrik P. Kingma, Danilo J. Rezende, Shakir

Semi-supervised Learning with Deep Generative Models Diedrik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling What is Deep Learning very good at? Classifying highly structured data -ImageNet -Part of Speech Tagging -MNIST

195 views • 18 slides
Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan

Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan

Introduction Variational Inference Deep Generative Models Summary Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan Fall 2015 Rahul G. Krishnan Learning Deep Generative Models Introduction

1.22k views • 83 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
FROM SCRATCH TO PRODUCTION Andrew Liu, Ryan Shen Deep Learning Solution Architect Defect

FROM SCRATCH TO PRODUCTION Andrew Liu, Ryan Shen Deep Learning Solution Architect Defect

DEFECT INSPECTION FROM SCRATCH TO PRODUCTION Andrew Liu, Ryan Shen Deep Learning Solution Architect Defect Inspection and its challenges NGC Docker images AGENDA Model set up - Unet Data preparation - DAGM Production - Speed up with

691 views • 33 slides
4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept.

4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept.

4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept. of Neuroradiology, Heidelberg University Hospital Div. of Medical Image Computing, DKFZ Heidelberg Faculty of Physics & Astronomy, Heidelberg

1.48k views • 65 slides
Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Universit Paris Fabien.Moutarde@mines-paristech.fr

777 views • 23 slides
Optimization for Training Deep Models presented by Kan Ren Table of Contents Optimization

Optimization for Training Deep Models presented by Kan Ren Table of Contents Optimization

Optimization for Training Deep Models presented by Kan Ren Table of Contents Optimization for machine learning models Challenges of optimizing neural networks Optimizations algorithms initializations adapting the learning

865 views • 62 slides
Deep Learning Models for Time Series Data Analysis with Applications to Health Care Yan Liu

Deep Learning Models for Time Series Data Analysis with Applications to Health Care Yan Liu

Deep Learning Models for Time Series Data Analysis with Applications to Health Care Yan Liu Computer Science Department University of Southern California Email: yanliu@usc.edu Yan Liu (USC) Deep Health 1 / 34 A human being is a part of a

515 views • 34 slides
a shallow survey of deep learning Applications, Models, Algorithms and Theory (?) Chiyuan Zhang

a shallow survey of deep learning Applications, Models, Algorithms and Theory (?) Chiyuan Zhang

a shallow survey of deep learning Applications, Models, Algorithms and Theory (?) Chiyuan Zhang April 22, 2015 CSAIL, Poggio Lab the state of the art the state of the art Deep learning is achieving the state of the art performance in many

606 views • 48 slides
CS7015 (Deep Learning) : Lecture 22 Autoregressive Models (NADE, MADE) Mitesh M. Khapra

CS7015 (Deep Learning) : Lecture 22 Autoregressive Models (NADE, MADE) Mitesh M. Khapra

CS7015 (Deep Learning) : Lecture 22 Autoregressive Models (NADE, MADE) Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/24 Mitesh M. Khapra CS7015 (Deep Learning) : Lecture 22 Module

980 views • 95 slides
CS7015 (Deep Learning) : Lecture 22 Autoregressive Models (NADE, MADE) Mitesh M. Khapra

CS7015 (Deep Learning) : Lecture 22 Autoregressive Models (NADE, MADE) Mitesh M. Khapra

CS7015 (Deep Learning) : Lecture 22 Autoregressive Models (NADE, MADE) Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/24 Mitesh M. Khapra CS7015 (Deep Learning) : Lecture 22 Module

772 views • 24 slides
Variational Inference and Generative Models CS 294-112: Deep Reinforcement Learning Sergey

Variational Inference and Generative Models CS 294-112: Deep Reinforcement Learning Sergey

Variational Inference and Generative Models CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 due next Wednesday 2. Accept CMT peer review invitations These are required (part of your final project grade)

652 views • 34 slides
Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

307 views • 26 slides

More recommend