Introduction to Machine Learning Hyperparameter Tuning - Basic - PowerPoint PPT Presentation

Feb 03, 2023 •970 likes •1.06k views

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques compstat-lmu.github.io/lecture_i2ml GRID SEARCH Simple technique which is still quite popular, tries all HP combinations on a multi-dimensional discretized grid For

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques compstat-lmu.github.io/lecture_i2ml
GRID SEARCH Simple technique which is still quite popular, tries all HP combinations on a multi-dimensional discretized grid For each hyperparameter a finite set of candidates is predefined Then, we simply search all possible combinations in arbitrary order Grid search over 10x10 points ● ● ● ● ● ● 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 TestAccuracy Hyperparameter 2 ● ● ● ● ● ● ● ● ● ● 0.8 ● ● ● ● ● ● ● ● ● ● 0.7 0 ● ● ● ● ● ● ● ● ● ● 0.6 0.5 ● ● ● ● ● ● ● ● ● ● −5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −10 ● ● ● ● ● ● ● ● ● ● −10 −5 0 5 10 Hyperparameter 1 � c Introduction to Machine Learning – 1 / 6
GRID SEARCH Advantages Very easy to implement All parameter types possible Parallelizing computation is trivial Disadvantages Scales badly: Combinatorial explosion Inefficient: Searches large irrelevant areas Arbitrary: Which values / discretization? � c Introduction to Machine Learning – 2 / 6
RANDOM SEARCH Small variation of grid search Uniformly sample from the region-of-interest Random search over 100 points ● ● ● ● ● ● ● ● 5 ● ● ● TestAccuracy Hyperparameter 2 ● ● ● 0.8 0 ● ● 0.7 ● ● ● ● ● ● ● 0.6 ● ● ● 0.5 ● ● −5 ● ● ● ● ● ● ● ● ● ● ● −10 ● −10 −5 0 5 10 Hyperparameter 1 � c Introduction to Machine Learning – 3 / 6
RANDOM SEARCH Advantages Like grid search: Very easy to implement, all parameter types possible, trivial parallelization Anytime algorithm: Can stop the search whenever our budget for computation is exhausted, or continue until we reach our performance goal. No discretization: each individual parameter is tried with a different value every time Disadvantages � c Introduction to Machine Learning – 4 / 6
RANDOM SEARCH Inefficient: many evaluations in areas with low likelihood for improvement Scales badly: high dimensional hyperparameter spaces need lots of samples to cover. � c Introduction to Machine Learning – 5 / 6
TUNING EXAMPLE Tuning random forest with random search and 5CV on the sonar data set for AUC: Parameter Type Min Max num.trees integer 3 500 mtry integer 5 50 integer 10 100 min.node.size Maximal AUC 0.93 0.92 0.91 0 50 100 150 Iterations � c Introduction to Machine Learning – 6 / 6

Recommend

Introduction to Machine Learning Hyperparameter Tuning - Introduction

Introduction to Machine Learning Hyperparameter Tuning - Introduction compstat-lmu.github.io/lecture_i2ml MOTIVATING EXAMPLE Given a dataset, we want to train a classification tree. We feel that a maximum tree depth of 4 has worked out well for

499 views • 7 slides

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition compstat-lmu.github.io/lecture_i2ml TUNING Recall: Hyperparameters are parameters that are inputs to the training problem, in which a learner I minimizes the

504 views • 7 slides

Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R What is H2O? library(h2o) h2o.init() H2O is not running yet, starting it now...

773 views • 30 slides

Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning in R The mlr package mlr is another framework for machine learning in R. Model

454 views • 28 slides

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data Patrick Schratz 1 , Jannes Muenchow 1 , Eugenia Iturritxa 2 , Jakob Richter 3 , Alexander Brenning 1 September 27 2018, 10th

632 views • 37 slides

Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R Voter dataset from US 2016 election Split intro training and test set

779 views • 22 slides

Introduction to Machine Learning Tuning: Nested Resampling Motivation

Introduction to Machine Learning Tuning: Nested Resampling Motivation compstat-lmu.github.io/lecture_i2ml MOTIVATION Selecting the best model from a set of potential candidates (e.g., different classes of learners, different hyperparameter

662 views • 9 slides

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data P a tr ic k S ch r a tz 1 , J a nn e s M u e n ch ow 1 , J a ko b R ich t e r 2 , A l e x a n de r B r e nn i n g 1 GIS cie n ce S e m i

688 views • 32 slides

Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor

Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor marc.claesen@esat.kuleuven.be

770 views • 57 slides

Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark

Deep Learning Hyperparameter Optimization with Competing Objectives GTC 2018 - S8136 Scott Clark scott@sigopt.com OUTLINE 1. Why is Tuning Models Hard? 2. Common Tuning Methods 3. Deep Learning Example 4. Tuning Multiple Metrics 5.

615 views • 43 slides

Hyperparameter Tuning in Python Using Optunity http://www.optunity.net Marc Claesen Jaak Simm

Introduction Optunity overview Example: tuning an SVM References Hyperparameter Tuning in Python Using Optunity http://www.optunity.net Marc Claesen Jaak Simm Dusan Popovic Bart De Moor ESAT-STADIUS, KU Leuven iMinds Medical IT Department

306 views • 29 slides

Asynchronous Hyperparameter Tuning and Ablation Studies with Apache Spark Sina Sheikholeslami

Asynchronous Hyperparameter Tuning and Ablation Studies with Apache Spark Sina Sheikholeslami Distributed Computing Group, KTH Royal Institute of Technology @cutlash CASTOR Software Days 2019 October 16 2019 sinash@kth.se The Machine

891 views • 43 slides

Parameters vs hyperparameters Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Parameters vs hyperparameters Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R About me www.shirin-glander.de DataCamp Hyperparameter Tuning in R

610 views • 24 slides

Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist Model parameters Parameters are: Learned or estimated from the data The result of tting a model Used when making future predictions Not

1.82k views • 35 slides

Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja S * 1, 2 Florian Mai

Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja S * 1, 2 Florian Mai * 1, 2 Thijs Vogels 2 Martin Jaggi 2 Franois Fleuret 1, 2 1 Idiap Research Institute, 2 EPFL, Switzerland * Equal Contribution prabhu.teja,

282 views • 15 slides

Introduction to Machine Learning Tuning: Nested Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Tuning: Nested Resampling compstat-lmu.github.io/lecture_i2ml NESTED RESAMPLING Just like we can generalize holdout splitting to resampling to get more reliable estimates of the predictive performance, we can

545 views • 7 slides

Automatic Database Management System Tuning Through Large-scale Machine Learning Dana Van Aken

Automatic Database Management System Tuning Through Large-scale Machine Learning Dana Van Aken Andrew Pavlo Geoffrey J. Gordon Bohan Zhang 1 Weakness in manually DBMS tuning Dependencies between different knobs Changing of

1.12k views • 22 slides

Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/ Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many slides attributable to: Prof. Mike Hughes Erik Sudderth

298 views • 28 slides

A Short Introduction to Bayesian Optimization With applications to parameter tuning on

A Short Introduction to Bayesian Optimization With applications to parameter tuning on accelerators Johannes Kirschner 28th February 2018 ICFA Workshop on Machine Learning for Accelerator Control Solve x = arg max f ( x ) x X 0

511 views • 36 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

565 views • 12 slides

Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and

Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and time-consuming Algorithm configuration using Machine Learning Focus on tree search algorithms Branch-and-Bound Tree Search Widely

396 views • 17 slides

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides

Automatic Database Management System Tuning Through Large-scale Machine Learning Van Aken Dana et

Automatic Database Management System Tuning Through Large-scale Machine Learning Van Aken Dana et al. [1] LSDPO (2017/2018) Paper Presentation: Ioana Bica (ib354) Standard approach: employ a database administrator (DBA) to tweak knobs

1.15k views • 38 slides

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Introduction to Machine Learning What is Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T

220 views • 20 slides