asynchronous hyperparameter tuning and ablation studies
play

Asynchronous Hyperparameter Tuning and Ablation Studies with Apache - PowerPoint PPT Presentation

Nov 11, 2022 •445 likes •891 views

Asynchronous Hyperparameter Tuning and Ablation Studies with Apache Spark Sina Sheikholeslami Distributed Computing Group, KTH Royal Institute of Technology @cutlash CASTOR Software Days 2019 October 16 2019 sinash@kth.se The Machine

  1. Asynchronous Hyperparameter Tuning and Ablation Studies with Apache Spark Sina Sheikholeslami Distributed Computing Group, KTH Royal Institute of Technology @cutlash CASTOR Software Days 2019 October 16 2019 sinash@kth.se

  2. The Machine Learning System Repeat if needed Machine Learning Dataset Model Problem Definition Data Preparation Evaluate Model Selection Optimizer Model Training October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 2

  3. Artificial Neural Networks Output Layer Input Layer Hidden Layer October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 3

  4. How We Study the Brain • Early 19 th Century, ablative brain surgeries by Jean Pierre Flourens (1794 - 1867) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 4

  5. Ablation for Machine Learning? floors area rooms price Repeat if needed Machine Learning Dataset Model Problem Definition Data Preparation Evaluate Model Selection Optimizer Model Training October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 5

  6. Talk of the Town “Too frequently, authors propose many tweaks absent proper ablation studies … Sometimes just one of the changes is actually responsible for the improved results … this practice misleads readers to believe that all of the proposed changes are necessary.” (Lipton & Steinhardt, “ Troubling Trends in Machine Learning Scholarship ”) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 6

  7. Example: Layer Ablation (1/6) Accuracy: 78% The Base Model October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 7

  8. Example: Layer Ablation (2/6) Accuracy: 73% October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 8

  9. Example: Layer Ablation (3/6) The Base Model October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 9

  10. Example: Layer Ablation (4/6) Accuracy: 67% October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 10

  11. Example: Layer Ablation (5/6) The Base Model October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 11

  12. Example: Layer Ablation (6/6) Accuracy: 63% October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 12

  13. Ablation Study Evaluate Machine Learning Ablation System New Dataset / Model Configuration October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 13

  14. Hyperparameter Tuning Evaluate Machine Learning Hyperparameter System Tuner New Hyperparameter Values October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 14

  15. System Experimentation (Search) Evaluate Global Machine Learning Experiment System Controller New Trial October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 15

  16. Better Parallel • Ability to train better models, faster • Ability to modify and inspect, easier (“Parallel Training” - by Maxim Melnikov) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 18

  17. Parallelization in Practice Machine Learning Parallel Deep Learning Processing (TensorFlow, the TensorFlow logo and any related marks are trademarks of Google Inc.) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 19

  18. Hopsworks Open-source Platform for Data-intensive AI October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 20

  19. Hopsworks Open-source Platform for Data-intensive AI What is Hopsworks? https://tinyurl.com/y4ze79d4 October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 21

  20. ML/DL in Hopsworks Machine Learning Experiments Data Pipelines Feature Data Parallel Model Serving Hyperparameter Ablation Ingest & Prep Store Training Tuning Studies Bottleneck, due to iterative nature • human interaction • October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 22

  21. Spark and Bulk Synchronous Parallel Model HDFS Metrics 3 Metrics 2 Metrics 1 Task 11 Task 21 Task 31 Task 12 Task 22 Task 32 Barrier Barrier Barrier Task 13 Task 23 Task 33 … … … Task 1N Task 2N Task 3N Driver October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 23

  22. Example: Synchronous Hyperparameter Search HDFS Metrics 3 Metrics 1 Metrics 2 Task 11 Task 21 Task 31 Task 12 Task 22 Task 32 Barrier Barrier Barrier Task 13 Task 23 Task 33 … … … Task 1N Task 2N Task 3N Wasted Compute Wasted Compute Wasted Compute Driver October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 24

  23. Critical Requirements • Parallel execution of trials • Support for early stopping of trials • Support for global control of the experiment • Resilience to stragglers • Simple, “Unified” User & Developer API October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 25

  24. Maggy An Open-source Framework for Asynchronous Computation on top of Apache Spark October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 26

  25. Key Idea: Long Running Tasks Task 11 Task 12 Barrier Task 13 Metrics … New Trial Task 1N Driver October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 27

  26. Maggy Core Architecture October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 28

  27. Back to Ablation October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 29

  28. LOCO: Leave One Component Out • A simple, “natural” ablation policy: an implementation of an ablator • Currently supports Feature Ablation + Layer Ablation October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 30

  29. Feature Ablation • Uses the Feature Store to access the dataset metadata • Generates Python callables that once called, will return modified datasets • Removes one-feature-at-a-time floors price floors rooms price area rooms October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 31

  30. Layer Ablation • Uses a base model function • Generates Python callables that once called, will return modified models • Uses the model configuration to find and remove layer(s) • Removes one-layer-at-a-time (or one-layer-group-at-a-time) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 32

  31. (Example Notebook Available!) Ablation User & Developer API October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 33

  32. User API: Initialize the Study and Add Features October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 34

  33. User API: Define Base Model October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 35

  34. User API: Setup Model Ablation October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 36

  35. User API: Wrap the Training Function October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 37

  36. User API: Lagom! October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 38

  37. Developer API: Policy Implementation (1/2) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 39

  38. Developer API: Policy Implementation (2/2) October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 40

  39. Hyperparameter Tuning: User API October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 41

  40. Hyperparameter Tuning: Developer API October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 42

  41. Maggy is Open-source • Code Repository: https://github.com/logicalclocks/maggy • API Documentation: https://maggy.readthedocs.io/en/latest/ October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 43

  42. Next Steps • More Ablators • More Tuners • Support for More Frameworks October 16 2019 Sina Sheikholeslami - KTH Royal Institute of Technology 44

  43. Thanks to the entire Logical Clocks Team J Specially: Thank you! J Moritz Meister @morimeister Jim Dowling @jim_dowling Robin Andersson @robzor92 Kim Hammar @KimHammar1 Alex Ormenisan @alex_ormenisan @logicalclocks @hopsworks GitHub (Example Notebook Available!) https://github.com/hopshadoop/maggy https://maggy.readthedocs.io/en/latest/ https://logicalclocks.com/whitepapers/ @cutlash sinash@kth.se October 16 2019 CASTOR Software Days 2019

Recommend

Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R Voter dataset from US 2016 election Split intro training and test set

779 views • 22 slides
Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R

Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R What is H2O? library(h2o) h2o.init() H2O is not running yet, starting it now...

773 views • 30 slides
Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning

Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning in R The mlr package mlr is another framework for machine learning in R. Model

454 views • 28 slides
Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition compstat-lmu.github.io/lecture_i2ml TUNING Recall: Hyperparameters are parameters that are inputs to the training problem, in which a learner I minimizes the

504 views • 7 slides
Parameters vs hyperparameters Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning

Parameters vs hyperparameters Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Parameters vs hyperparameters Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R About me www.shirin-glander.de DataCamp Hyperparameter Tuning in R

610 views • 24 slides
Hyperparameter Tuning in Python Using Optunity http://www.optunity.net Marc Claesen Jaak Simm

Hyperparameter Tuning in Python Using Optunity http://www.optunity.net Marc Claesen Jaak Simm

Introduction Optunity overview Example: tuning an SVM References Hyperparameter Tuning in Python Using Optunity http://www.optunity.net Marc Claesen Jaak Simm Dusan Popovic Bart De Moor ESAT-STADIUS, KU Leuven iMinds Medical IT Department

306 views • 29 slides
Introduction to Machine Learning Hyperparameter Tuning - Introduction

Introduction to Machine Learning Hyperparameter Tuning - Introduction

Introduction to Machine Learning Hyperparameter Tuning - Introduction compstat-lmu.github.io/lecture_i2ml MOTIVATING EXAMPLE Given a dataset, we want to train a classification tree. We feel that a maximum tree depth of 4 has worked out well for

499 views • 7 slides
Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data Patrick Schratz 1 , Jannes Muenchow 1 , Eugenia Iturritxa 2 , Jakob Richter 3 , Alexander Brenning 1 September 27 2018, 10th

632 views • 37 slides
Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques compstat-lmu.github.io/lecture_i2ml GRID SEARCH Simple technique which is still quite popular, tries all HP combinations on a multi-dimensional discretized grid For

1.06k views • 7 slides
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja S * 1, 2 Florian Mai

Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja S * 1, 2 Florian Mai

Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja S * 1, 2 Florian Mai * 1, 2 Thijs Vogels 2 Martin Jaggi 2 Franois Fleuret 1, 2 1 Idiap Research Institute, 2 EPFL, Switzerland * Equal Contribution prabhu.teja,

282 views • 15 slides
Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark

Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark

Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark scott@sigopt.com OUTLINE 1. Why is Tuning Models Hard? 2. Common Tuning Methods 3. Deep Learning Example 4. Tuning Multiple Metrics 5.

615 views • 43 slides
Maggy - Open-Source Asynchronous Distributed Hyperparameter Optimization Based on Apache Spark

Maggy - Open-Source Asynchronous Distributed Hyperparameter Optimization Based on Apache Spark

Maggy - Open-Source Asynchronous Distributed Hyperparameter Optimization Based on Apache Spark Moritz Meister FOSDEM20 moritz@logicalclocks.com 02.02.2020 @morimeister The Bitter Lesson (of AI)* Methods that scale with computation

924 views • 30 slides
Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data P a tr ic k S ch r a tz 1 , J a nn e s M u e n ch ow 1 , J a ko b R ich t e r 2 , A l e x a n de r B r e nn i n g 1 GIS cie n ce S e m i

688 views • 32 slides
Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist Model parameters Parameters are: Learned or estimated from the data The result of tting a model Used when making future predictions Not

1.82k views • 35 slides
Transformer Ablation Studies Simon Will Institute of Formal and Applied Linguistics Charles

Transformer Ablation Studies Simon Will Institute of Formal and Applied Linguistics Charles

. . . . . . . . . . . . . . . . Transformer Ablation Studies Simon Will Institute of Formal and Applied Linguistics Charles University Seminar: Statistical Machine Translation Instructor: Dr. Ondej Bojar . . . . . . .

980 views • 19 slides
Introduction to Machine Learning Tuning: Nested Resampling Motivation

Introduction to Machine Learning Tuning: Nested Resampling Motivation

Introduction to Machine Learning Tuning: Nested Resampling Motivation compstat-lmu.github.io/lecture_i2ml MOTIVATION Selecting the best model from a set of potential candidates (e.g., different classes of learners, different hyperparameter

662 views • 9 slides
Asynchronous Replication and Bayou Asynchronous Replication and Bayou Asynchronous Replication

Asynchronous Replication and Bayou Asynchronous Replication and Bayou Asynchronous Replication

Asynchronous Replication and Bayou Asynchronous Replication and Bayou Asynchronous Replication Asynchronous Replication Idea: build available/scalable information services with read-any-write-any replication and a weak consistency model. - no

738 views • 48 slides
L ABLATION DANS LES ABLATION DANS LES L ARYTHMIES ARYTHMIES SUPRAVENTRICULAIRES

L ABLATION DANS LES ABLATION DANS LES L ARYTHMIES ARYTHMIES SUPRAVENTRICULAIRES

L ABLATION DANS LES ABLATION DANS LES L ARYTHMIES ARYTHMIES SUPRAVENTRICULAIRES SUPRAVENTRICULAIRES PROF L DE ROY UNIVERSITE DE LOUVAIN BELGIQUE AV NODE ABLATION AV NODE ABLATION RF ON ACCESSORY PATHWAYS ACCESSORY PATHWAYS RF

1.22k views • 43 slides
Hyperparameter optimization strategies git clone

Hyperparameter optimization strategies git clone

Hyperparameter optimization strategies git clone https://github.com/IASIAI/hyperparameter-optimization-strategies.git meetup.com/ IASI-AI / facebook.com/ AI.Iasi / iasiai.net Gabriel Marchidan Bogdan Burlacu Software architect AI researcher ,

456 views • 29 slides
THERMAL ABLATION TATO SYSTEM Laura Sarti Product Specialist 1 Definition Thermal Ablation (TA)

THERMAL ABLATION TATO SYSTEM Laura Sarti Product Specialist 1 Definition Thermal Ablation (TA)

THERMAL ABLATION TATO SYSTEM Laura Sarti Product Specialist 1 Definition Thermal Ablation (TA) is the use of extreme hiperthermic temperatures to destroy diseased tissues. T.A. is performed in Interventional Radiology as an alternative to the

547 views • 50 slides
Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor

Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor

Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor marc.claesen@esat.kuleuven.be

770 views • 57 slides
Disclosure Ablation and Devices for Atrial Fibrillation SentreHeart/Atricure, Inc Should all

Disclosure Ablation and Devices for Atrial Fibrillation SentreHeart/Atricure, Inc Should all

10/10/19 Disclosure Ablation and Devices for Atrial Fibrillation SentreHeart/Atricure, Inc Should all Patients have Ablation? 1 2 AF PAROXYSMAL PERSISTENT AF LONG STANDING AF . PERMANENT AF . Diagnosis AF Risk Factor Modification 33.5

422 views • 9 slides
NO CONFLICT OF Associate Chief, Cardiac Electrophysiology Director, Cardiac Electrophysiology

NO CONFLICT OF Associate Chief, Cardiac Electrophysiology Director, Cardiac Electrophysiology

Mapping and ablation of Junctional Mapping and ablation of Junctional Mapping and ablation of Junctional Mapping and ablation of Junctional Tachycardias Tachycardias Tachycardias Tachycardias Nitish Badhwar, MD, FACC, FHRS NO CONFLICT OF

363 views • 14 slides
Asynchronous sequence circuits An asynchronous sequence machine is a sequence circuit without

Asynchronous sequence circuits An asynchronous sequence machine is a sequence circuit without

Asynchronous sequence circuits An asynchronous sequence machine is a sequence circuit without flip-flops Asynchronous sequence machines are based on combinational gates with feedback Upon analysis it is assumed : Only one signal at a

1.34k views • 66 slides

More recommend