Bringing the human back in the loop Joaquin Vanschoren, TU/e (new) - PowerPoint PPT Presentation

Bringing the human back in the loop Joaquin Vanschoren, TU/e (new) data Meta-learning Automate optimization data analysis improved initial workflows workflows Online collaboration

Automating machine learning: a human-robot symbiosis Joaquin Vanschoren, TU/e (new) data Meta-learning Robot optimization assistants OpenML datasets, workflows, meta-data improved initial meta-models workflows workflows Online collaboration

Robot assistants Data type bot: detects/assigns correct data types removes unique/constant features Missing value bot: detects miscoded missing values, correlation with target (new) data Anomaly detection (WTF) bot: spurious strings, data artifacts DataDiff bot: detects data changes (data type, statistical deviations, …) initial workflows Thanks to Rich Caruana

Robot assistants Data similarity bot: finds datasets similar to yours (meta-features) Label imbalance bot: detects/ reduces class imbalance (e.g. SMOTE) Encoding bot: converts to numeric data (new) data depending on ML algo (SVM, kNN, NN) Data leak bot: detects if test data leaks into the training set (e.g. by inspecting initial workflow, withholding data) workflows

Robot assistants Runtime prediction bot: predicts how long an ML algorithm will run on your data Feature selection bot: recommends/runs feature selection techniques (new) data Imputation bot: recommends/runs missing value imputation techniques Outlier detection bot: recommends/ runs outlier detection techniques initial workflows

Robot assistants Random Bot: runs random search given a hyperparameter space Greedy Bot: learns key algorithms, hyper- parameters, ranges. Tries those first. (new) data Optimization bots: runs advanced hyperparameter optimization Workflow bot: build ML workflows, in collaboration with other bots initial workflows

Random Bot on OpenML

Publish workflows on OpenML for collaboration Joaquin Vanschoren, TU/e (new) data Robot assistants OpenML datasets, workflows, meta-data improved initial meta-models workflows workflows Online collaboration

Learn from results of humans and robots, use that to build better bots (new) data Meta-learning Robot optimization assistants OpenML datasets, workflows, meta-data improved initial meta-models workflows workflows Online collaboration

Learn from results of humans and robots, -> build better bots, more insight Data similarity bot: Meta-features across workflow Runtime prediction bot: many runtime results Feature selection bot: how do different techniques affect learning performance? Optimization bots: use meta-data to search hyperparameter space more effectively

Combine meta-learning and optimization Bayesian optimization • Warm start: initialize search with promising configurations • Transfer learning, e.g. surrogate models with prior • Acquisition functions based on meta-models

Combine meta-learning and optimization • Acquisition functions based on meta-models

Combine meta-learning and optimization (Adaptive) multi-armed bandits • Use meta-learning to predict which configurations to race first • Learn from previous iterations • Active testing: choose configurations that outperformed the surviving configurations on similar datasets.

Thank you (new) data Meta-learning Automate optimization data analysis improved initial workflows workflows Online collaboration