Hyperparameter Tuning in Python Using Optunity - PowerPoint PPT Presentation

Introduction Optunity overview Example: tuning an SVM References Hyperparameter Tuning in Python Using Optunity http://www.optunity.net Marc Claesen Jaak Simm Dusan Popovic Bart De Moor ESAT-STADIUS, KU Leuven iMinds Medical IT Department September 9, 2014 STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Outline Introduction 1 Optunity overview 2 Example: tuning an SVM 3 Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Hyperparameter tuning Tuning is the task of obtaining hyperparameters for a given learning approach that yield a good model for given data. occurs in both supervised and unsupervised learning hyperparameter choice can significantly impact performance Some examples: SVM: regularization and kernel hyperparameters ANN: regularization and network architecture KSC: number of clusters and kernel hyperparameters Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Tuning in practice Most often done using a combination of grid and manual search: grid search suffers from the curse of dimensionality promotes ad hoc tuning approaches Manual tuning leads to poor reproducibility requires expert knowledge a black art (Snoek, Larochelle, & Adams, 2012) Better solutions exist but lack adoption because: potential performance improvements are underestimated lack of availability and/or ease of use Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) In optimization terms, we aim to find λ ∗ (assuming minimization): λ ∗ = arg min A ( X ( tr ) | λ ) | X ( te ) � � L λ Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) In optimization terms, we aim to find λ ∗ (assuming minimization): λ ∗ = arg min A ( X ( tr ) | λ ) | X ( te ) � � F ( λ | A , X ( tr ) , X ( te ) ) L = arg min � �� � λ λ objective function Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Outline Introduction 1 Optunity overview 2 Example: tuning an SVM 3 Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Design goals Our key goals are: providing appropriate solvers for the tuning task terse user and dev APIs with easy learning curves compatibility with any version of Python ( > 2 . 7) easy integration within alien environments Optunity’s implementation is not focused on efficiency. evaluating F is the performance bottleneck external dependencies are avoided (NumPy, scikit-learn, . . . ) We are all consenting adults. Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References General approach Optunity solves tuning problems using: methods for continuous black-box function optimization sequential function evaluations (with vectorization support) expert and simplified programming interfaces (optional) domain constraints on the objective function F Using the simplified interface entails specifying: F and the permitted number of evaluations box constraints for each hyperparameter (optional) the solver to be used Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Solvers We offer a variety of fundamentally different approaches to be suitable for many problems (Wolpert & Macready, 1997). The following solvers are currently available: undirected search grid search random search (Bergstra & Bengio, 2012) Nelder-Mead simplex (Nelder & Mead, 1965) evolutionary methods (using DEAP (Fortin et al., 2012)) particle swarm optimization (Kennedy, 2010) CMA-ES (Hansen & Ostermeier, 2001) Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Cross-validation Optunity provides k-fold cross-validation to estimate performance of supervised algorithms implemented as a function decorator Configurable features: number of folds and iterations static or dynamic folds support for strata and clusters Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Integrating Optunity in alien environments Optunity can be integrated in other environments easily: launchable as standalone subprocess no language bindings necessary communication using JSON messages through pipes minimal data exchange (e.g. no data sets) arbitrary method ⇔ JSON ⇔ generic solvers R   grid search working environment Optunity in Python     MATLAB random search       Java Wrapper Nelder-Mead Method API Solvers C/C ++ particle swarm     � callback requests ⋆     . . . CMA-ES  �  configuration ⋆ callback results final solution Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Current development Implementing additional solvers: coupled simulated annealing (Xavier-de Souza, Suykens, Vandewalle, & Boll´ e, 2010) efficient global optimization (Jones, Schonlau, & Welch, 1998) other Bayesian approaches (Hutter, Hoos, & Leyton-Brown, 2011) Integrating Optunity in alien environments: nearly finished: R and MATLAB planned: Java and C/C++ Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Outline Introduction 1 Optunity overview 2 Example: tuning an SVM 3 Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction S T A D I U S Optunity overview Example: tuning an SVM Center for Dynamical Systems, Signal Processing and Data Analytics References Task summary Tuning an SVM classifier with RBF kernel κ ( u , v ) = e − γ � u − v � 2 : optimize regularization parameter C and kernel parameter γ evaluate ( C , γ ) pair using 2 × iterated 10-fold cross-validation Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity