hyperparameter search in machine learning
play

Hyperparameter Search in Machine Learning Marc Claesen and Bart De - PowerPoint PPT Presentation

May 26, 2023 •186 likes •770 views

Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor marc.claesen@esat.kuleuven.be

  1. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor marc.claesen@esat.kuleuven.be ESAT-STADIUS, KU Leuven iMinds Medical IT Department STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  2. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Outline Introduction 1 Example: optimizing hyperparameters for an SVM classifier 2 Challenges in hyperparameter search 3 State-of-the-art 4 Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  3. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Machine learning Methods capable of learning patterns of interest from data. by formulating the learning task as an optimization problem Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  4. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Machine learning Methods capable of learning patterns of interest from data. by formulating the learning task as an optimization problem Machine learning is situated on the intersection of various fields: statistics, computer science, optimization, (biology), . . . Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  5. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Machine learning Methods capable of learning patterns of interest from data. by formulating the learning task as an optimization problem Machine learning is situated on the intersection of various fields: statistics, computer science, optimization, (biology), . . . The field encompasses learning methods with various origins, e.g.: biology, e.g. neural networks [1] convex optimization, e.g. support vector machines [2] statistics, e.g. hidden Markov models [3] tensor decompositions, e.g. recommender systems [4] Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  6. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter search Most machine learning methods are (hyper)parameterized. e.g. Occam’s razor: model complexity and overfitting Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  7. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter search Most machine learning methods are (hyper)parameterized. e.g. Occam’s razor: model complexity and overfitting Hyperparameters can significantly impact performance suitable hyperparameters must be determined for each task occurs in both supervised and unsupervised learning → need for disciplined, automated optimization methods Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  8. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter search Most machine learning methods are (hyper)parameterized. e.g. Occam’s razor: model complexity and overfitting Hyperparameters can significantly impact performance suitable hyperparameters must be determined for each task occurs in both supervised and unsupervised learning → need for disciplined, automated optimization methods Some examples: SVM: regularization and kernel hyperparameters ANN: regularization, network architecture, transfer functions Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  9. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  10. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  11. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  12. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  13. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) In optimization terms, we aim to find λ ∗ (assuming minimization): λ ∗ = arg min A ( X ( tr ) | λ ) | X ( te ) � � L λ Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  14. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) In optimization terms, we aim to find λ ∗ (assuming minimization): λ ∗ = arg min A ( X ( tr ) | λ ) | X ( te ) � � F ( λ | A , X ( tr ) , X ( te ) , L ) L = arg min � �� � λ λ objective function Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  15. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Tuning in practice Most often done using a combination of grid and manual search: grid search suffers from the curse of dimensionality manual tuning leads to poor reproducibility Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  16. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Tuning in practice Most often done using a combination of grid and manual search: grid search suffers from the curse of dimensionality manual tuning leads to poor reproducibility Better solutions exist but lack adoption because: potential performance improvements are underestimated lack of availability and/or ease of use Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  17. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Outline Introduction 1 Example: optimizing hyperparameters for an SVM classifier 2 Challenges in hyperparameter search 3 State-of-the-art 4 Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  18. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Support vector machine (SVM) classifiers Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  19. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Support vector machine (SVM) classifiers Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

  20. Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Support vector machine (SVM) classifiers n 1 � � � min α i α j y i y j κ ( x i , x j ) + C ξ i , 2 α,ξ, b i =1 i ∈ SV j ∈ SV � � � α i α j y i y j κ ( x i , x j ) + b ≥ 1 − ξ i , ξ i ≥ 0 , ∀ i . subject to y i j ∈ SV Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning

Recommend

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques

Introduction to Machine Learning Hyperparameter Tuning - Basic Techniques compstat-lmu.github.io/lecture_i2ml GRID SEARCH Simple technique which is still quite popular, tries all HP combinations on a multi-dimensional discretized grid For

1.06k views • 7 slides
Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning

Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Machine learning with mlr Dr. Shirin Elsinghorst Data Scientist DataCamp Hyperparameter Tuning in R The mlr package mlr is another framework for machine learning in R. Model

454 views • 28 slides
Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R

Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Machine learning with H2O Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R What is H2O? library(h2o) h2o.init() H2O is not running yet, starting it now...

773 views • 30 slides
Introduction to Machine Learning Hyperparameter Tuning - Introduction

Introduction to Machine Learning Hyperparameter Tuning - Introduction

Introduction to Machine Learning Hyperparameter Tuning - Introduction compstat-lmu.github.io/lecture_i2ml MOTIVATING EXAMPLE Given a dataset, we want to train a classification tree. We feel that a maximum tree depth of 4 has worked out well for

499 views • 7 slides
Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data Patrick Schratz 1 , Jannes Muenchow 1 , Eugenia Iturritxa 2 , Jakob Richter 3 , Alexander Brenning 1 September 27 2018, 10th

632 views • 37 slides
Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition compstat-lmu.github.io/lecture_i2ml TUNING Recall: Hyperparameters are parameters that are inputs to the training problem, in which a learner I minimizes the

504 views • 7 slides
Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data P a tr ic k S ch r a tz 1 , J a nn e s M u e n ch ow 1 , J a ko b R ich t e r 2 , A l e x a n de r B r e nn i n g 1 GIS cie n ce S e m i

688 views • 32 slides
Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R Voter dataset from US 2016 election Split intro training and test set

779 views • 22 slides
Introduction to Machine Learning Tuning: Nested Resampling Motivation

Introduction to Machine Learning Tuning: Nested Resampling Motivation

Introduction to Machine Learning Tuning: Nested Resampling Motivation compstat-lmu.github.io/lecture_i2ml MOTIVATION Selecting the best model from a set of potential candidates (e.g., different classes of learners, different hyperparameter

662 views • 9 slides
Example 1 : Selecting a present for somebody You enter a large shop with no idea what

Example 1 : Selecting a present for somebody You enter a large shop with no idea what

Evolutionary Computing (Genetic Algorithms) Heralded as an approach to machine learning Learning new solutions/behaviours by evolving old ones Also an approach to search Search in a potentially infinite search space No

321 views • 6 slides
Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/ Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many slides attributable to: Prof. Mike Hughes Erik Sudderth

298 views • 28 slides
Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t

Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t

Akinori Tanaka (RIKEN AIP/iTHEMS) Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t machine learning OR deep learning and date 20xx->20xx+1 year number of results Intro Mainly experimental, a few

908 views • 70 slides
Leveraging AWS and Machine Learning to Power Search at Zocdoc Pedro Rubio Head of Search

Leveraging AWS and Machine Learning to Power Search at Zocdoc Pedro Rubio Head of Search

Leveraging AWS and Machine Learning to Power Search at Zocdoc Pedro Rubio Head of Search Engineering Brian dAlessandro Head of Data Science This document and its contents are proprietary and confidential of Zocdoc, Inc. and may not be

582 views • 35 slides
A majorization-minimization algorithm for (multiple) hyperparameter learning Chuan-Sheng Foo

A majorization-minimization algorithm for (multiple) hyperparameter learning Chuan-Sheng Foo

A majorization-minimization algorithm for (multiple) hyperparameter learning Chuan-Sheng Foo Chuong B. Do Andrew Y. Ng Stanford University ICML 2009 Montreal, Canada 17 th June 2009 Supervised learning Training set of m IID examples

221 views • 21 slides
Learning to Search + Recurrent Neural Networks Matt Gormley Lecture 4 Sep. 9, 2019 1

Learning to Search + Recurrent Neural Networks Matt Gormley Lecture 4 Sep. 9, 2019 1

10-418 / 10-618 Machine Learning for Structured Data Machine Learning Department School of Computer Science Carnegie Mellon University Learning to Search + Recurrent Neural Networks Matt Gormley Lecture 4 Sep. 9, 2019 1 Reminders

599 views • 58 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

559 views • 17 slides
Probabilistic & Unsupervised Learning Model selection, Hyperparameter optimisation, and

Probabilistic & Unsupervised Learning Model selection, Hyperparameter optimisation, and

Probabilistic & Unsupervised Learning Model selection, Hyperparameter optimisation, and Gaussian Processes Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University

1.03k views • 84 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

401 views • 26 slides
CMSC5743 L09: Network Architecture Search Bei Yu (Latest update: September 13, 2020) Fall 2020

CMSC5743 L09: Network Architecture Search Bei Yu (Latest update: September 13, 2020) Fall 2020

CMSC5743 L09: Network Architecture Search Bei Yu (Latest update: September 13, 2020) Fall 2020 1 / 29 Overview Search Space Design Blackbox Optimization NAS as a hyperparameter optimization Reinforcement Learning Evolution methods

567 views • 33 slides
10a Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10a Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10a Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

935 views • 54 slides
10d Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10d Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10d Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

1.05k views • 29 slides
10b Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10b Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10b Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

866 views • 50 slides

More recommend