machine learning and discovery with kubernetes
play

Machine learning and discovery with Kubernetes William Benton - PowerPoint PPT Presentation

Jan 31, 2024 •133 likes •907 views

Machine learning and discovery with Kubernetes William Benton @willb willb@redhat.com What do machine learning workflows look like? @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19

  1. Machine learning and discovery with Kubernetes William Benton • @willb • willb@redhat.com

  2. What do machine learning workflows look like?

  3. @willb #SEMLA19

  4. @willb #SEMLA19

  5. @willb #SEMLA19

  6. @willb #SEMLA19

  7. @willb #SEMLA19

  8. @willb #SEMLA19

  9. @willb #SEMLA19

  10. codifying problem 
 data collection and metrics and cleaning @willb #SEMLA19

  11. data collection feature model training and cleaning engineering and tuning @willb #SEMLA19

  12. data collection feature model training and cleaning engineering and tuning @willb #SEMLA19

  13. feature model training model engineering and tuning validation @willb #SEMLA19

  14. feature model training model engineering and tuning validation @willb #SEMLA19

  15. model model monitoring and validation deployment validation @willb #SEMLA19

  16. model model monitoring and validation deployment validation @willb #SEMLA19

  17. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation @willb #SEMLA19

  18. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation defining types prototyping and interfaces @willb #SEMLA19

  19. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation unit, behavioral, and formal integration testing verification @willb #SEMLA19

  20. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation deployment monitoring @willb #SEMLA19

  21. @willb #SEMLA19

  22. @willb #SEMLA19

  23. What’s a container?

  24. % @willb #SEMLA19

  25. % pip install numpy @willb #SEMLA19

  26. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  27. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  28. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  29. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  30. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  31. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem /var/lib/envs/main process table network routes @willb #SEMLA19

  32. executable /usr/bin/pip arguments pip install numpy MAXIMUM environment LANG=en_US USER=willb ... 100 virtual memory file handles root filesystem /var/lib/envs/main process table km / h network routes @willb #SEMLA19

  33. Immutable images a6afd91e user application code 6b8cad3e 33721112 configuration and e8cae4f6 2bb6ab16 installation recipes a8296f7e base image 979229b9 @willb #SEMLA19

  34. Stateless microservices @willb #SEMLA19

  35. Stateless microservices @willb #SEMLA19

  36. Stateless microservices @willb #SEMLA19

  37. Stateless microservices @willb #SEMLA19

  38. Stateless microservices @willb #SEMLA19

  39. Stateless microservices @willb #SEMLA19

  40. Stateless microservices @willb #SEMLA19

  41. Stateless microservices @willb #SEMLA19

  42. Declarative app configuration @willb #SEMLA19

  43. Integration and deployment OK! @willb #SEMLA19

  44. Integration and deployment application code OK! configuration and installation recipes base image @willb #SEMLA19

  45. Integration and deployment application code OK! configuration and installation recipes base image @willb #SEMLA19

  46. Integration and deployment application code configuration and installation recipes base image @willb #SEMLA19

  47. What containers offer 
 data scientists

  48. @willb #SEMLA19

  49. @willb #SEMLA19

  50. O K ! O K ! @willb #SEMLA19

  51. No friction: mybinder.org @willb #SEMLA19

  52. More flexible: source-to-image % @willb #SEMLA19

  53. More flexible: source-to-image % builder image application image https://github.com/openshift/source-to-image @willb #SEMLA19

  56. @willb #SEMLA19

  57. @willb #SEMLA19

  60. @willb #SEMLA19

  61. m A @willb #SEMLA19

  62. m A @willb #SEMLA19

  63. (joint) distribution of input data? distribution of predictions? m A distribution of acyclic paths taken through scoring code? @willb #SEMLA19

  64. @willb #SEMLA19

  65. Where from here?

  66. application developers data engineers web and transform developer UI events mobile transform federate databases archive file, object transform reporting storage train management models data scientists @willb #SEMLA19

  67. application developers data engineers web and transform developer UI events mobile transform federate databases archive file, object transform reporting storage train management models machine learning engineers data scientists @willb #SEMLA19

  68. radanalytics.io @willb #SEMLA19

  69. opendatahub.io @willb #SEMLA19

  70. Kubeflow @willb #SEMLA19

  71. What did we talk about today?

  72. @willb #SEMLA19

  73. @willb #SEMLA19

  74. @willb #SEMLA19

  75. @willb #SEMLA19

  76. THANKS willb@redhat.com • @willb https://chapeau.freevariable.com @willb #SEMLA19

Recommend

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining Aufgaben Aufgabe Von Daten zum Wissen Wissensextraktion durch We are drowing in automatisches Erkennen von information,

1.01k views • 47 slides
Machine-Learning & AI-based Approaches for GPCR Bioactive Ligand Discovery Sebastian Raschka,

Machine-Learning & AI-based Approaches for GPCR Bioactive Ligand Discovery Sebastian Raschka,

Cambridge Healthtech Institute's 14th Annual GPCR-Based Drug Discovery Discovery on Target Conference September 19, 2019 | Boston, MA https://www.discoveryontarget.com/GPCR-drug-discovery Machine-Learning & AI-based Approaches for GPCR

411 views • 38 slides
Kubernetes for machine learning: productivity over primitives Sophie Watson @sophwats

Kubernetes for machine learning: productivity over primitives Sophie Watson @sophwats

Kubernetes for machine learning: productivity over primitives Sophie Watson @sophwats sophie@redhat.com William Benton @willb willb@redhat.com Kubernetes for machine learning: productivity over primitives Sophie

1.24k views • 102 slides
Machine Learning for Information Discovery Thorsten Joachims Cornell University Department of

Machine Learning for Information Discovery Thorsten Joachims Cornell University Department of

Machine Learning for Information Discovery Thorsten Joachims Cornell University Department of Computer Science (Supervised) Machine Learning GENERAL: EXAMPLE: Text Retrieval Input: Input: training examples queries with relevance

700 views • 28 slides
D ata-driven drug discovery for a variety of diseases by machine learning

D ata-driven drug discovery for a variety of diseases by machine learning

D ata-driven drug discovery for a variety of diseases by machine learning Yoshihiro Yamanishi Kyushu University Medical Ins;tute of Bioregula;on

833 views • 25 slides
Edge-based Discovery of Training Data for Machine Learning CMU authors Deep Learning Recipe

Edge-based Discovery of Training Data for Machine Learning CMU authors Deep Learning Recipe

CSci 8980 Edge-based Discovery of Training Data for Machine Learning CMU authors Deep Learning Recipe Collect a large amount of data and label it Select a model and train a DNN Deploy the DNN for inference Labelled Data Some

409 views • 14 slides
Discovery of Latent Factors in High-dimensional Data via Spectral Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data via Spectral Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data via Spectral Methods Furong Huang University of Maryland Workshop on Quantum Machine Learning 1 / 39 Machine Learning - Excitements Success of Supervised Learning Image classification

1.24k views • 80 slides
Airflow on Kubernetes: Containerizing your Workflows By Michael Hewitt Agenda Kubernetes

Airflow on Kubernetes: Containerizing your Workflows By Michael Hewitt Agenda Kubernetes

Airflow on Kubernetes: Containerizing your Workflows By Michael Hewitt Agenda Kubernetes Overview 1 Airflows integration with Kubernetes 2 Deployment of Airflow on Kubernetes 3 Kubernetes Pod Operator and its benefits 4 DAG Development

2.79k views • 14 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
with Kubernetes and Tensorflow Daniel van der Ende & Tim van Cann o IT Consultancy o 40

with Kubernetes and Tensorflow Daniel van der Ende & Tim van Cann o IT Consultancy o 40

Data Council NYC 2019 Reducing Flight Delays with Kubernetes and Tensorflow Daniel van der Ende & Tim van Cann o IT Consultancy o 40 Data Scientists, Machine Learning Engineers, and Data Engineers o Based in Amsterdam o We help

885 views • 56 slides
Discovery of Latent Factors in High-dimensional Data Using Tensor Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data Using Tensor Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data Using Tensor Methods Furong Huang University of California, Irvine Machine Learning Conference 2016 New York City 1 / 24 Machine Learning - Modern Challenges Big Data Challenging Tasks

686 views • 57 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

581 views • 5 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Mitul Tiwari and Deepak Bobbarjung

TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Mitul Tiwari and Deepak Bobbarjung

TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Mitul Tiwari and Deepak Bobbarjung Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework on Kubernetes Our Jobs architecture Our Conversational AI Platform

595 views • 30 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
Home Learning Event NSS Discovery: National benchmarking February 2020 What is NSS Discovery?

Home Learning Event NSS Discovery: National benchmarking February 2020 What is NSS Discovery?

Hospital at Home Learning Event NSS Discovery: National benchmarking February 2020 What is NSS Discovery? NSS Discovery is a management information system that provides approved users with access to a range of comparative health

579 views • 8 slides
Roll your own Service Discovery Simon Pasquier @SimonHiker Prometheus Service Discovery Avoid

Roll your own Service Discovery Simon Pasquier @SimonHiker Prometheus Service Discovery Avoid

Roll your own Service Discovery Simon Pasquier @SimonHiker Prometheus Service Discovery Avoid static configuration files No reloading Always up-to-date Currently supported Files DNS Consul Kubernetes Marathon

819 views • 14 slides
Continuous Kubernetes Security @sublimino and @controlplaneio Im: - Andy - Dev-like -

Continuous Kubernetes Security @sublimino and @controlplaneio Im: - Andy - Dev-like -

Continuous Kubernetes Security @sublimino and @controlplaneio Im: - Andy - Dev-like - Sec-ish - Ops-y Is this Kubernetes cluster secure? EPIC FAIL GIF How secure is Kubernetes? What this Kubernetes talk is about Common Pwns

1.18k views • 113 slides
Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine Learning? What is Machine Learning? Examples of machine learning applications: Facial recognition software E-mail spam detectors Automated

302 views • 17 slides

More recommend