cross validation ensembling
play

Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw - PowerPoint PPT Presentation

Jul 16, 2023 •226 likes •1.18k views

Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 1 / 34 Outline Cross

  1. Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 1 / 34

  2. Outline Cross Validation 1 How Many Folds? Ensemble Methods 2 Voting Bagging Boosting Why AdaBoost Works? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 2 / 34

  3. Outline Cross Validation 1 How Many Folds? Ensemble Methods 2 Voting Bagging Boosting Why AdaBoost Works? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 3 / 34

  4. Cross Validation So far, we use the hold out method for: Hyperparameter tuning: validation set Performance reporting: testing set What if we get an “unfortunate” split? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 4 / 34

  5. Cross Validation So far, we use the hold out method for: Hyperparameter tuning: validation set Performance reporting: testing set What if we get an “unfortunate” split? K -fold cross validation : Split the data set X evenly into K subsets X ( i ) (called folds ) 1 For i = 1 , ··· , K , train f � N ( i ) using all data but the i -th fold ( X \ X ( i ) ) 2 Report the cross-validation error C CV by averaging all testing errors 3 C [ f � N ( i ) ] ’s on X ( i ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 4 / 34

  6. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  7. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Inner (2 folds): select 1 hyperparameters giving lowest C CV Can be wrapped by grid search Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  8. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Inner (2 folds): select 1 hyperparameters giving lowest C CV Can be wrapped by grid search Train final model using both 2 training and validation sets with the selected hyperparameters Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  9. Nested Cross Validation Cross validation (CV) can be applied to both hyperparameter tuning and performance reporting E.g, 5 ⇥ 2 nested CV Inner (2 folds): select 1 hyperparameters giving lowest C CV Can be wrapped by grid search Train final model using both 2 training and validation sets with the selected hyperparameters Outer (5 folds): report C CV as 3 test error Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 5 / 34

  10. Outline Cross Validation 1 How Many Folds? Ensemble Methods 2 Voting Bagging Boosting Why AdaBoost Works? Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 6 / 34

  11. How Many Folds K ? I Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  12. How Many Folds K ? I The cross-validation error C CV is an average of C [ f � N ( i ) ] ’s Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  13. How Many Folds K ? I The cross-validation error C CV is an average of C [ f � N ( i ) ] ’s Regard each C [ f � N ( i ) ] as an estimator of the expected generalization error E X ( C [ f N ]) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  14. How Many Folds K ? I The cross-validation error C CV is an average of C [ f � N ( i ) ] ’s Regard each C [ f � N ( i ) ] as an estimator of the expected generalization error E X ( C [ f N ]) C CV is an estimator too, and we have MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 7 / 34

  15. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  16. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  17. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  18. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) � ˆ ( ˆ θ n � E [ ˆ ( E [ ˆ θ n � E [ ˆ ( E [ ˆ ⇥ θ n ]) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ � = E + E + 2E θ n ] θ n ] � θ ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  19. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) � ˆ ( ˆ θ n � E [ ˆ ( E [ ˆ θ n � E [ ˆ ( E [ ˆ ⇥ θ n ]) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ � = E + E + 2E θ n ] θ n ] � θ ) � 2 + 2 · 0 · ( E [ ˆ ( ˆ θ n � E [ ˆ E [ ˆ ⇥ θ n ]) 2 ⇤ � = E + θ n ] � θ θ n ] � θ ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  20. Point Estimation Revisited: Mean Square Error Let ˆ θ n be an estimator of quantity θ related to random variable x mapped from n i.i.d samples of x Mean square error of ˆ θ n : MSE ( ˆ ( ˆ θ n � θ ) 2 ⇤ ⇥ θ n ) = E X Can be decomposed into the bias and variance: ( ˆ ( ˆ θ n � E [ ˆ θ n ]+ E [ ˆ ⇥ θ n � θ ) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ = E E X θ n ]) 2 +( E [ ˆ θ n ] � θ ) 2 + 2 ( ˆ ( ˆ θ n � E [ ˆ θ n � E [ ˆ θ n ])( E [ ˆ ⇥ ⇤ = E θ n ] � θ ) � ˆ ( ˆ θ n � E [ ˆ ( E [ ˆ θ n � E [ ˆ ( E [ ˆ ⇥ θ n ]) 2 ⇤ ⇥ θ n ] � θ ) 2 ⇤ � = E + E + 2E θ n ] θ n ] � θ ) � 2 + 2 · 0 · ( E [ ˆ ( ˆ θ n � E [ ˆ E [ ˆ ⇥ θ n ]) 2 ⇤ � = E + θ n ] � θ θ n ] � θ ) = Var X ( ˆ θ n )+ bias ( ˆ θ n ) 2 MSE of an unbiased estimator is its variance Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 8 / 34

  21. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  22. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  23. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Let C [ · ] be the MSE of predictions (made by a function) to true labels Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  24. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Let C [ · ] be the MSE of predictions (made by a function) to true labels E X ( C [ f N ]) : read line Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

  25. Example: 5-Fold vs. 10-Fold CV MSE ( C CV ) = E X [( C CV � E X ( C [ f N ])) 2 ] = Var X ( C CV )+ bias ( C CV ) 2 Consider polynomial regression where P ( y | x ) = sin ( x )+ ε , ε ⇠ N ( 0 , σ 2 ) Let C [ · ] be the MSE of predictions (made by a function) to true labels E X ( C [ f N ]) : read line bias ( C CV ) : gaps between the red and other solid lines ( E X [ C CV ] ) Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning 9 / 34

Recommend

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides
Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Classier evaluation Classier evaluation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Classier evaluation Classier evaluation Leave-one-out Cross-Validation Resampled validation set

984 views • 51 slides
in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

Accelerating Cross-Validation in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101 [Wikipedia] Popular Model Validation Technique to avoid overfitting, for better generalization useful when not

430 views • 20 slides
Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation for Covariate Shift (1) (2) Masashi Sugiyama , Benjamin Blankertz , (2,3) (2) Matthias Krauledat , Guido Dornhege , (3,2) Klaus-Robert

623 views • 26 slides
Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013 TIN2010-20900-C04 Albacete Index Introduction Multilabel Classification Cross-Validation and Stratified Cross-Validation Methods and

682 views • 43 slides
Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method Transfer, partial and cross validation Team members: In scope Life cycle of a method after first full validation or relation Team

252 views • 10 slides
Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. These methods refit a model of interest to samples formed from the training set, in order to obtain additional

688 views • 44 slides
Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Outline Background and Motivation Classifier Ensembling Genetic Algorithms Proposed Method of Classifier Ensemble Feature Set Used Experimental Results Conclusions Future Works Maximum Entropy Classifier Ensembling using Ge- netic

749 views • 36 slides
STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

Outline Last Time Cross-Validation STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12 April 2016 Outline Last Time Cross-Validation Outline Last Time Cross-Validation Outline Last Time

381 views • 25 slides
Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks Criticality experiments and benchmarks for for validation of cross validation of cross sections: the neptunium sections: the neptunium case case L.S.Leong, L.Tassan-Got, L.Audouin, C. Paradela,

464 views • 21 slides
CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic* csc411ta@cs.toronto.edu *Based on the tutorial given by Erin Grant, Ziyu Zhang, and Ali Punjani in previous years. Outline for Today Cross-Validation

355 views • 24 slides
Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides
15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J. Zico Kolter Carnegie Mellon University Fall 2019 1 Outline Example: return to peak demand prediction Overfitting, generalization, and cross

903 views • 67 slides
Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views • 58 slides
Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides
From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high throughput data compiling a complete collection of transcription factor binding motifs Vsevolod J. Makeev Vavilov Institute of General Genetics,

721 views • 37 slides
Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy

Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy

Approximate Cross-Validation and Dynamic Experiments for Policy Choice Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy Department of Economics, Harvard University April 23, 2018 1 / 23 Approximate

588 views • 23 slides
Holdout and Cross- -Validation Validation Holdout and Cross Methods Overfitting Avoidance

Holdout and Cross- -Validation Validation Holdout and Cross Methods Overfitting Avoidance

Holdout and Cross- -Validation Validation Holdout and Cross Methods Overfitting Avoidance Methods Overfitting Avoidance Decision Trees Decision Trees Reduce error pruning Reduce error pruning Cost Cost- -complexity pruning

699 views • 17 slides
Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. 1 / 44 Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the

1.13k views • 66 slides
Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications

Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications

Higher Order Analysis of Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications IV In honor of Prof. Amaris 80 th birthday June 12-17, 2016, Liblice, Czech Republic Sumio Watanabe Tokyo Institute of

486 views • 30 slides
Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using given data What makes a model Features Hyper-parameters that control the hypothesis space Example: depth of a decision tree, neural

328 views • 12 slides
Significance testing after cross-validation Joshua Loftus ( jloftus@turing.ac.uk ) (building from

Significance testing after cross-validation Joshua Loftus ( jloftus@turing.ac.uk ) (building from

Significance testing after cross-validation Joshua Loftus ( jloftus@turing.ac.uk ) (building from joint work with Jonathan Taylor) 9 December, 2016 Slides and markdown source at https://joftius.github.io/turing 1 / 20 Setting: regression model

590 views • 23 slides
Evaluate Deep Q-Learning for Sequential Targeted Marketing with 10-fold Cross Validation

Evaluate Deep Q-Learning for Sequential Targeted Marketing with 10-fold Cross Validation

Agent Actions Evaluate Deep Q-Learning for Sequential Targeted Marketing with 10-fold Cross Validation Rewards Observations Jian Wu Full Stack Engineer Marketing Environment The idea of this talk was developed at Samsung SDSRA AI Lab

208 views • 19 slides
5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1

5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1

5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1 Computer Laboratory University of Cambridge Lent 2018 1 Based on slides created by Simone Teufel Last session: smoothing and significance testing

1.56k views • 15 slides

More recommend