Maggy - Open-Source Asynchronous Distributed Hyperparameter - - PowerPoint PPT Presentation

Moritz Meister moritz@logicalclocks.com @morimeister

Maggy

• Open-Source Asynchronous Distributed

Hyperparameter Optimization Based on Apache Spark

FOSDEM’20 02.02.2020

The Bitter Lesson (of AI)*

“Methods that scale with computation are the future of AI”** “The two (general purpose) methods that seem to scale … ... are search and learning.”*

2

** https://www.youtube.com/watch?v=EeMCEQa85tw

Rich Sutton

(Father of Reinforcement Learning)

* http://www.incompleteideas.net/IncIdeas/BitterLesson.html

The Answer Spark scales with available compute! is the answer!

Distribution and Deep Learning

Generalization Error Better Regularization Methods Design Better Models Hyperparameter Optimization Larger Training Datasets Better Optimization Algorithms Distribution

Inner Loop Outer Loop worker1

Inner and Outer Loop of Deep Learning

worker2 workerN

…

∆

2

∆

N

Synchronization

http://tiny.cc/51yjdz

Training Data

∆

1

Metric

Search Method

Trial HParams, Architecture, etc.

Inner Loop Outer Loop worker1

Inner and Outer Loop of Deep Learning

worker2 workerN

…

∆

2

∆

N

Synchronization

http://tiny.cc/51yjdz

Training Data

∆

1

Metric

SEARCH LEARNING

Search Method

Trial HParams, Architecture, etc.

In Reality This Means Rewriting Training Code

Data Pipelines Ingest & Prep Feature Store Machine Learning Experiments Data Parallel Training Model Serving Ablation Studies Hyperparameter Optimization Explore/ Design Model

Towards Distribution Transparency

The distribution oblivious training function (pseudo-code):

Inner Loop

Towards Distribution Transparency

• Trial and Error is slow
• Iterative approach is greedy
• Search spaces are usually large
• Sensitivity and interaction of

hyperparameters

Set Hyper- parameters Train Model Evaluate Performance

Sequential Black Box Optimization

Learning Black Box Metric Meta-level learning &

• ptimization

Search space

Outer Loop

Sequential Search

Learning Black Box Metric Global Controller

Outer Loop

Parallel Search

Learning Black Box Metric Global Controller Parallel Workers Queue

Trial Trial

Search space or Ablation Study Which algorithm to use for search? How to monitor progress? Fault Tolerance? How to aggregate results?

Outer Loop

Parallel Search

Learning Black Box Metric Meta-level learning &

• ptimization

Parallel Workers Queue

Trial Trial

Search space Which algorithm to use for search? How to monitor progress? Fault Tolerance? How to aggregate results?

This should be managed with platform support!

Maggy

A flexible framework for asynchronous parallel execution of trials for ML experiments on Hopsworks: ASHA, Random Search, Grid Search, LOCO-Ablation, Bayesian Optimization and more to come…

Synchronous Search

Task11

Driver

Task12 Task13 Task1N

…

HDFS

Task21 Task22 Task23 Task2N

…

Barrier Barrier Task31 Task32 Task33 Task3N

…

Barrier

Metrics1 Metrics2 Metrics3

Add Early Stopping and Asynchronous Algorithms

Task11

Driver

Task12 Task13 Task1N

…

HDFS

Task21 Task22 Task23 Task2N

…

Barrier Barrier Task31 Task32 Task33 Task3N

…

Barrier

Metrics1 Metrics2 Metrics3 Wasted Compute Wasted Compute Wasted Compute

Performance Enhancement

Early Stopping:

• Median Stopping Rule
• Performance curve prediction

Multi-fidelity Methods:

• Successive Halving Algorithm
• Hyperband

Asynchronous Successive Halving Algorithm

Animation: https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/ Liam Li et al. “Massively Parallel Hyperparameter Tuning”. In: CoRR abs/1810.05934 (2018).

Ablation Studies

PClass name survive sex sex name survive

Replacing the Maggy Optimizer with an Ablator:

• Feature Ablation using

the Feature Store

• Leave-One-Layer-Out

Ablation

• Leave-One-Component-Out

(LOCO)

How can we fit this into the bulk synchronous execution model of Spark? Mismatch: Spark Tasks and Stages vs. Trials

Challenge

Databricks’ approach: Project Hydrogen (barrier execution mode) & SparkTrials in Hyperopt

The Solution

Task11

Driver

Task12 Task13 Task1N

…

Barrier

Metrics New Trial Early Stop

Long running tasks and communication:

HyperOpt: One Job/Trial, requiring many Threads on Driver

Enter Maggy

User API

Developer API

Ablation API

Ablation API

Results

Hyperparameter Optimization Task ASHA Validation Task ASHA RS-ES RS-NS ASHA RS-ES RS-NS

Conclusions

• Avoid iterative Hyperparameter Optimization
• Black box optimization is hard
• State-of-the-art algorithms can be deployed

asynchronously

• Maggy: platform support for automated hyperparameter
• ptimization and ablation studies
• Save resources with asynchronism
• Early stopping for sensible models

What’s next?

• More algorithms
• Distribution

Transparency

• Comparability/

reproducibility of experiments

• Implicit Provenance
• Support for PyTorch

Acknowledgements

Thanks to the entire Logical Clocks Team ☺ Contributions from colleagues: Robin Andersson @robzor92 Sina Sheikholeslami @cutlash Kim Hammar @KimHammar1 Alex Ormenisan @alex_ormenisan

• Maggy

https://github.com/logicalclocks/maggy https://maggy.readthedocs.io/en/latest/

• Hopsworks

https://github.com/logicalclocks/hopsworks https://www.logicalclocks.com/whitepapers/hopsworks

• Feature Store: the missing data layer in ML pipelines?

https://www.logicalclocks.com/feature-store/

@hopsworks HOPSWORKS