lecture 7 cross validation
play

Lecture 7: Cross-Validation Instructor: Prof. Shuai Huang - PowerPoint PPT Presentation

Dec 23, 2022 β€’504 likes β€’640 views

Lecture 7: Cross-Validation Instructor: Prof. Shuai Huang Industrial and Systems Engineering University of Washington Underfit, Good fit, and Overfit 2 = 0 + 1 1 + 2 2 + 11 1 2

  1. Lecture 7: Cross-Validation Instructor: Prof. Shuai Huang Industrial and Systems Engineering University of Washington

  2. Underfit, Good fit, and Overfit 𝑔 π’š 𝑔 π’š 2 = 𝛾 0 + 𝛾 1 𝑦 1 + 𝛾 2 𝑦 2 + 𝛾 11 𝑦 1 2 𝑔 π’š = 𝛾 0 + 𝛾 1 𝑦 1 + 𝛾 2 𝑦 2 = 𝛾 0 + 𝛾 1 𝑦 1 + 𝛾 2 𝑦 2 + 𝛾 11 𝑦 1 2 + 𝛾 12 𝑦 1 𝑦 2 + 𝛾 112 𝑦 1 2 𝑦 2 + 𝛾 22 𝑦 2 2 + 𝛾 12 𝑦 1 𝑦 2 + 𝛾 22 𝑦 2 2 + β‹― + 𝛾 122 𝑦 1 𝑦 2

  3. Danger of R-squared β€’ When number of variables increases, in theory, the R- squared won’t decrease; in practice, it always increases. Thus, it is not a good metric to take into consideration of model complexity 𝑆 2 = 1 βˆ’ 𝑇𝑇𝐹 π‘‡π‘‡π‘ˆ β€’ This is because that: ST is always fixed, while SSE could only decrease if more variables are put into the model even if these new added variables have no relationship with the outcome variable

  4. Danger of R- squared (cont’d) β€’ Further, the R-squared is compounded by the variance of predictors as well. As the underlying regression model is 𝑍 = π›Ύπ‘Œ + πœ— , β€’ The variance of 𝑍 , 𝑀𝑏𝑠 𝑍 = 𝛾 2 𝑀𝑏𝑠 π‘Œ + 𝑀𝑏𝑠(πœ—) . The R-squared takes the form as 𝛾 2 𝑀𝑏𝑠 π‘Œ R-squared= 𝛾 2 𝑀𝑏𝑠 π‘Œ +𝑀𝑏𝑠(πœ—) . β€’ Thus, it seems that R-squared is not only impacted by how well π‘Œ can predict 𝑍 , but also by the variance of π‘Œ as well.

  5. The truth about training error β€’ Just as the R-squared, it will continue to decrease if the model is mathematically more complex (therefore, more able to shape itself to make its prediction correct on data points that are due to noise)

  6. Fix R-squared: AIC/BIC/? IC… β€’ The definition of AIC (Akaike Information Criterion) 𝐡𝐽𝐷 = 2𝑙 βˆ’ 2 ln ΰ·  𝑀 β€’ The definition of BIC (Bayesian Information Criterion) 𝐢𝐽𝐷 = ln 𝑂 𝑙 βˆ’ 2 ln ΰ·  𝑀

  7. Training and testing data β€’ A simple strategy: if a model is good, then it should perform well on an unseen testing data (that represents the future data – which is of course unseen in the model training stage)

  8. K-Fold cross-validation β€’ For example, K=4

  9. Random sampling method β€’ How to conduct the training/testing data scheme, when we only have access to a dataset (usually we take this dataset as β€œtraining data” – a concept taken for granted)?

  10. Other dimensions of β€œerror” β€’ The TP, FP, FN, TN

  11. The ROC curve (Receiver Operating Characteristics) β€’ Consider a logistic regression model

  12. R lab β€’ Download the markdown code from course website β€’ Conduct the experiments β€’ Interpret the results β€’ Repeat the analysis on other datasets

Recommend

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views β€’ 15 slides
Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views β€’ 58 slides
Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views β€’ 31 slides
Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Classier evaluation Classier evaluation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Classier evaluation Classier evaluation Leave-one-out Cross-Validation Resampled validation set

984 views β€’ 51 slides
in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

Accelerating Cross-Validation in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101 [Wikipedia] Popular Model Validation Technique to avoid overfitting, for better generalization useful when not

430 views β€’ 20 slides
Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation for Covariate Shift (1) (2) Masashi Sugiyama , Benjamin Blankertz , (2,3) (2) Matthias Krauledat , Guido Dornhege , (3,2) Klaus-Robert

623 views β€’ 26 slides
Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013 TIN2010-20900-C04 Albacete Index Introduction Multilabel Classification Cross-Validation and Stratified Cross-Validation Methods and

682 views β€’ 43 slides
Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Review Multiple Regression with Interaction Terms Model Selection:

522 views β€’ 33 slides
Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method Transfer, partial and cross validation Team members: In scope Life cycle of a method after first full validation or relation Team

252 views β€’ 10 slides
Machine Learning Lecture 7 Some feature engineering and Cross validation Justin Pearson 1 2020 1

Machine Learning Lecture 7 Some feature engineering and Cross validation Justin Pearson 1 2020 1

Machine Learning Lecture 7 Some feature engineering and Cross validation Justin Pearson 1 2020 1 http://user.it.uu.se/~justin/Teaching/MachineLearning/index.html 1 / 35 Over Fitting vs Bias 4.0 3.8 3.6 3.4 3.2 3.0 0.00 0.25 0.50 0.75

720 views β€’ 35 slides
Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. These methods refit a model of interest to samples formed from the training set, in order to obtain additional

688 views β€’ 44 slides
STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

Outline Last Time Cross-Validation STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12 April 2016 Outline Last Time Cross-Validation Outline Last Time Cross-Validation Outline Last Time

381 views β€’ 25 slides
Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks Criticality experiments and benchmarks for for validation of cross validation of cross sections: the neptunium sections: the neptunium case case L.S.Leong, L.Tassan-Got, L.Audouin, C. Paradela,

464 views β€’ 21 slides
CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic* csc411ta@cs.toronto.edu *Based on the tutorial given by Erin Grant, Ziyu Zhang, and Ali Punjani in previous years. Outline for Today Cross-Validation

355 views β€’ 24 slides
Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views β€’ 55 slides
15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J. Zico Kolter Carnegie Mellon University Fall 2019 1 Outline Example: return to peak demand prediction Overfitting, generalization, and cross

903 views β€’ 67 slides
Lecture 9: Nonparametric Regression (1) Applied Statistics 2015 1 / 22 Introduction

Lecture 9: Nonparametric Regression (1) Applied Statistics 2015 1 / 22 Introduction

Introduction Estimations: local modelling Cross Validation Assignments Lecture 9: Nonparametric Regression (1) Applied Statistics 2015 1 / 22 Introduction Estimations: local modelling Cross Validation Assignments An example: Pick-It

459 views β€’ 25 slides
CSE 154 LECTURE 11: REGULAR EXPRESSIONS What is form validation? validation : ensuring that

CSE 154 LECTURE 11: REGULAR EXPRESSIONS What is form validation? validation : ensuring that

CSE 154 LECTURE 11: REGULAR EXPRESSIONS What is form validation? validation : ensuring that form's values are correct some types of validation: preventing blank values (email address) ensuring the type of values integer, real

920 views β€’ 25 slides
CSc 337 LECTURE 23: REGULAR EXPRESSIONS What is form validation? validation : ensuring that

CSc 337 LECTURE 23: REGULAR EXPRESSIONS What is form validation? validation : ensuring that

CSc 337 LECTURE 23: REGULAR EXPRESSIONS What is form validation? validation : ensuring that form's values are correct some types of validation: preventing blank values (email address) ensuring the type of values integer, real

524 views β€’ 20 slides
From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high throughput data compiling a complete collection of transcription factor binding motifs Vsevolod J. Makeev Vavilov Institute of General Genetics,

721 views β€’ 37 slides
Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy

Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy

Approximate Cross-Validation and Dynamic Experiments for Policy Choice Approximate Cross-Validation and Dynamic Experiments for Policy Choice Maximilian Kasy Department of Economics, Harvard University April 23, 2018 1 / 23 Approximate

588 views β€’ 23 slides
Holdout and Cross- -Validation Validation Holdout and Cross Methods Overfitting Avoidance

Holdout and Cross- -Validation Validation Holdout and Cross Methods Overfitting Avoidance

Holdout and Cross- -Validation Validation Holdout and Cross Methods Overfitting Avoidance Methods Overfitting Avoidance Decision Trees Decision Trees Reduce error pruning Reduce error pruning Cost Cost- -complexity pruning

699 views β€’ 17 slides
BBM406 Fundamentals of Machine Learning Lecture 6: Learning theory Probability Review Aykut

BBM406 Fundamentals of Machine Learning Lecture 6: Learning theory Probability Review Aykut

BBM406 Fundamentals of Machine Learning Lecture 6: Learning theory Probability Review Aykut Erdem // Hacettepe University // Fall 2019 Last time Regularization , Cross-Validation error Validation error the data Training error number

1.01k views β€’ 48 slides
Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. 1 / 44 Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the

1.13k views β€’ 66 slides

More recommend