learning algorithm evaluation outline
play

Learning Algorithm Evaluation Outline Why ? Overfitting How? - PowerPoint PPT Presentation

May 12, 2023 •195 likes •1.01k views

Disease + - t l TP FP + TP+FP u s e r FN TN t FN+TN - s e t TP+FN FP+TN Learning Algorithm Evaluation Outline Why ? Overfitting How? Holdout vs Cross-validation What? Evaluation measures Who wins?

  1. Disease + - t l TP FP + TP+FP u s e r FN TN t FN+TN - s e t TP+FN FP+TN Learning Algorithm Evaluation

  2. Outline Why ? Overfitting • How? Holdout vs Cross-validation • What? Evaluation measures • Who wins? Statistical significance •

  3. Quiz Is this a good model?

  4. Overfitting While it fits the training data perfectly, it may perform badly on unseen data. A simpler model may be better.

  5. Outline Why ? Overfitting • How? Holdout vs Cross-validation • What? Evaluation measures • Who wins? Statistical significance •

  6. A first evaluation measure • Predictive accuracy • Success : instance’s class is predicted correctly • Error : instance’s class is predicted incorrectly • Error rate : #errors/#instances • Predictive Accuracy : #successes/#instances • Quiz • 50 examples, 10 classified incorrectly • Accuracy? Error rate?

  7. Rule #1

  8. Rule #1 Never evaluate on training data!

  9. Holdout (Train and Test)

  10. Holdout (Train and Test)

  11. Holdout (Train and Test) a.k.a. holdout set

  12. Holdout (Train and Test) a.k.a. holdout set

  13. Holdout (Train and Test) a.k.a. holdout set

  14. Quiz Can I retry with other parameter settings?

  15. Rule #2

  16. Rule #2 Never train/optimize on test data! (that includes parameter selection)

  17. Holdout (Train and Test) You need a separate optimization set to tune parameters OPTIMIZATION TESTING

  18. Test data leakage Never use test data to create the classifier • Can be tricky: e.g. social network • Proper procedure uses three sets • training set : train models • optimization/validation set : optimize algorithm • parameters test set : evaluate final model •

  19. Holdout (Train and Test) Build final model on ALL data (more data, better model)

  20. Making the most of data • Once evaluation is complete, and algorithm/ parameters are selected, all the data can be used to build the final classifier • Trade-off: performance <-> evaluation accuracy More training data, better model (but returns diminish) • More test data, more accurate error estimate •

  21. Issues • Small data sets Random test set can be quite different from training set • (different data distribution) • Unbalanced class distributions One class can be overrepresented in test set • Serious problem for some domains: • medical diagnosis: 90% healthy, 10% disease • eCommerce: 99% don’t buy, 1% buy • Security: >99.99% of Americans are not terrorists •

  22. Balancing unbalanced data Sample equal amounts from minority and majority class + ensure approximately equal proportions in train/test set

  23. Stratified Sampling Advanced class balancing: sample so that each class represented with approx. equal proportions in both subsets E.g. take a stratified sample of 50 instances:

  24. Repeated holdout method Evaluation still biased by random test sample • Solution: repeat and average results • Random, stratified sampling, N times • Final performance = average of all performances •

  25. Repeated holdout method Evaluation still biased by random test sample • Solution: repeat and average results • Random, stratified sampling, N times • Final performance = average of all performances •

  26. Repeated holdout method Evaluation still biased by random test sample • Solution: repeat and average results • Random, stratified sampling, N times • Final performance = average of all performances • TRAIN TEST TRAIN TEST TRAIN TEST

  27. Repeated holdout method Evaluation still biased by random test sample • Solution: repeat and average results • Random, stratified sampling, N times • Final performance = average of all performances • TRAIN 0.86 TEST TRAIN 0.74 TEST TRAIN 0.8 TEST

  28. Repeated holdout method Evaluation still biased by random test sample • Solution: repeat and average results • Random, stratified sampling, N times • Final performance = average of all performances • TRAIN 0.86 0.8 TEST TRAIN 0.74 TEST TRAIN 0.8 TEST

  29. k-fold Cross-validation Split data (stratified) in k-folds Use (k-1) for training, 1 for testing, repeat k times, average results

  30. Cross-validation Standard method: • stratified 10-fold cross-validation • Experimentally determined. Removes most of • sampling bias Even better: repeated stratified cross-validation • Popular: 10 x 10-fold CV, 2 x 3-fold CV •

  31. Leave-One-Out Cross-validation A particular form of cross-validation: • #folds = #instances • n instances, build classifier n times • Makes best use of the data, no sampling bias • Computationally very expensive •

  32. Outline Why ? Overfitting • How? Holdout vs Cross-validation • What? Evaluation measures • Who wins? Statistical significance •

  33. Some other Evaluation Measures • ROC: Receiver-Operator Characteristic • Precision and Recall • Cost-sensitive learning • Evaluation for numeric predictions • MDL principle and Occam’s razor

  34. ROC curves ROC curves • • Receiver Operating Characteristic • From signal processing: tradeoff between hit rate and false alarm rate over noisy channel • Method: • Plot True Positive rate against False Positive rate

  35. Confusion Matrix actual + - TP FP + d e t true positive false positive c i TN FN d - e r false negative true negative p FP+TN TP+FN TPrate (sensitivity): FPrate (fall-out):

  36. ROC curves ROC curves • • Receiver Operating Characteristic • From signal processing: tradeoff between hit rate and false alarm rate over noisy channel • Method: • Plot True Positive rate against False Positive rate • Collect many points by varying prediction threshold • For probabilistic algorithms (probabilistic predictions) • Non-probabilistic algorithms have single point • Or, make cost sensitive and vary costs (see below)

  37. ROC curves Predictions actually actually inst P(+) actual positive negative 1 0.8 + 2 0.5 - FP 3 0.45 + 4 0.3 - TP FP

  38. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  39. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  40. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  41. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  42. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  43. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  44. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  45. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  46. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

  47. ROC curves Predictions Thresholds actually actually 0 0.3 0.45 0.5 0.8 inst P(+) actual positive negative 1 0.8 + + + + + - 2 0.5 - + + + - - FP 3 0.45 + + + - - - 4 0.3 - + - - - - TP 0 0.3 0.45 0.5 0.8 1 1 1/2 1/2 0 TPrate 1 1/2 1/2 0 0 FPrate FP

Recommend

Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of theoretical studies of deep learning 2. Optimization theory of deep neural networks 1) Gradient finds global optima 2) Gram-Gauss-Newton Algorithm

484 views • 31 slides
Supervised Learning Supervised learning algorithms require the presence of a teacher who

Supervised Learning Supervised learning algorithms require the presence of a teacher who

Backpropagation Learning Algorithm An algorithm for learning the weights in the network, given a training set of input-output pairs { , } The algorithm is based on gradient descent method. Architecture : Weight on

303 views • 9 slides
Evaluation for Strategic Learning London Funders, 10 February 2015 Session outline UK

Evaluation for Strategic Learning London Funders, 10 February 2015 Session outline UK

Evaluation for Strategic Learning London Funders, 10 February 2015 Session outline UK Evaluation Roundtable State of evaluation across the Roundtable network Evaluation for strategic learning Roundtable findings Discussion Use

572 views • 21 slides
Evaluation learning algorithm ? Do you want to predict accuracy or predict Charles Sutton

Evaluation learning algorithm ? Do you want to predict accuracy or predict Charles Sutton

Evaluate what? Do you want to evaluate a classifier or a Evaluation learning algorithm ? Do you want to predict accuracy or predict Charles Sutton which one is better ? Data Mining and Exploration Do you have a lot of data or not much

545 views • 17 slides
Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml RESAMPLING Aim: Assess the performance of learning algorithm. Make training sets large (to keep the pessimistic bias small), and reduce variance

778 views • 10 slides
PRELIMINARY PERFORMANCE EVALUATION OF HEF SCHEDULING ALGORITHM Carlos A. Rincn C. and Albert

PRELIMINARY PERFORMANCE EVALUATION OF HEF SCHEDULING ALGORITHM Carlos A. Rincn C. and Albert

PRELIMINARY PERFORMANCE EVALUATION OF HEF SCHEDULING ALGORITHM Carlos A. Rincn C. and Albert M. K. Cheng University of Houston Real-time Systems Lab carincon@uh.edu and cheng@cs.uh.edu OUTLINE Motivation Related work Methods and

292 views • 8 slides
11/27/2006 Massachusetts Institute of Technology Outline Motivation Example Generalized

11/27/2006 Massachusetts Institute of Technology Outline Motivation Example Generalized

11/27/2006 Massachusetts Institute of Technology Outline Motivation Example Generalized Conflict Learning for Problem Formulation Hybrid Discrete/Linear Optimization The GCD-BB algorithm Empirical Evaluation Conclusion

252 views • 4 slides
Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization

Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization

CS6501: T opics in Learning and Game Theory (Fall 2019) Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization Measure Algorithm Performance via Regret Warm-up: A Simple Example 2 Overview of Machine

629 views • 37 slides
Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA Other Views of Learning Is Learning Feasible: A Puzzle M. Magdon-Ismail CSCI 4100/6100 recap: The Plan 1. What is Learning? 2. Can We do it?

613 views • 25 slides
LEARNING Outline Data Buckets Processing Running an ML Algorithm Creating a

LEARNING Outline Data Buckets Processing Running an ML Algorithm Creating a

Python and AWS CSCI 447/547 MACHINE LEARNING Outline Data Buckets Processing Running an ML Algorithm Creating a Notebook Editing the Notebook Python Output Evaluating Results Getting Started in AWS Services

594 views • 22 slides
Signal Processing Algorithm Description &amp; Evaluation with Simulation Brooke Russell Yale

Signal Processing Algorithm Description &amp; Evaluation with Simulation Brooke Russell Yale

Signal Processing Algorithm Description &amp; Evaluation with Simulation Brooke Russell Yale University DUNE APA Consortium Meeting October 2 nd 2017 Outline Last meeting Hanyu summarized single phase LArTPC signal formation and a full

722 views • 23 slides
CSCI 447/547 MACHINE LEARNING Outline Nearest Neighbor K-Nearest Neighbor Algorithm

CSCI 447/547 MACHINE LEARNING Outline Nearest Neighbor K-Nearest Neighbor Algorithm

Nearest Neighbor CSCI 447/547 MACHINE LEARNING Outline Nearest Neighbor K-Nearest Neighbor Algorithm Note: Slides were adapted from David Sontag, New York University (who adapted them from Vibhav Gogate, Carlos Questrin, Mehryar

200 views • 16 slides
Lifelong Learning in Optimisation Emma Hart Edinburgh Napier University

Lifelong Learning in Optimisation Emma Hart Edinburgh Napier University

Lifelong Learning in Optimisation Emma Hart Edinburgh Napier University http://jamesobrien.tumblr.com/post/1112777561/lifel ong-learning-illustration Optimisation Algorithms Algorithm Algorithm Algorithm Algorithm Algorithm

625 views • 40 slides
1 The DFG Algorithm - The DFG Algorithm - Rationale The basic cycle Obviously, for each

1 The DFG Algorithm - The DFG Algorithm - Rationale The basic cycle Obviously, for each

4.3. Reinforcement learning for forming coalitions: the DFG algorithm Reinforcement Learning (I) Wei (1995) Watkins (1989) DFG: Dissolution and Formation of Groups Lets use our single agent definition: Then an agent Ag has in Dat for each

58 views • 3 slides
A Direct Policy-Search Algorithm for Relational Reinforcement Learning Samuel Sarjant Bernhard

A Direct Policy-Search Algorithm for Relational Reinforcement Learning Samuel Sarjant Bernhard

Introduction CERRLA Evaluation Conclusion and Remarks A Direct Policy-Search Algorithm for Relational Reinforcement Learning Samuel Sarjant Bernhard Pfahringer, Kurt Driessens, Tony Smith Department of Computer Science University of

537 views • 24 slides
The EM Algorithm The EM algorithm Mixture models Why EM works EM variants Learning

The EM Algorithm The EM algorithm Mixture models Why EM works EM variants Learning

Preview The EM Algorithm The EM algorithm Mixture models Why EM works EM variants Learning with Missing Data The EM Algorithm Goal: Learn parameters of Bayes net with known structure Initialize parameters ignoring missing

363 views • 3 slides
Learning Selection Strategies in Buchbergers Algorithm Dylan Peifer Cornell University 31

Learning Selection Strategies in Buchbergers Algorithm Dylan Peifer Cornell University 31

Learning Selection Strategies in Buchbergers Algorithm Dylan Peifer Cornell University 31 October 2019 Outline The efficiency of Buchbergers algorithm strongly depends on a choice of selection strategy. By phrasing Buchbergers

965 views • 80 slides
10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise, Listwise 10.3. Gathering User Input 10.4. LeToR Evaluation 10.5. Beyond Search Advanced Topics in Information Retrieval / Learning to Rank 2 10.1.

1.01k views • 34 slides
Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and

Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and

Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and Feedback for Instructors SELFI Initial USAT Charge Phase 1 USAT committee (2014 15) Adopt a new set of quantitative (Likert style) questions

610 views • 31 slides
Evaluation of training Learning objectives Design a possible evaluation procedure for

Evaluation of training Learning objectives Design a possible evaluation procedure for

INF3280, 15 Feb 2017 Evaluation of training Learning objectives Design a possible evaluation procedure for Assignment 5 Core literature Chapter 9 Auxiliary literature Holton III (1996) The flawed four-level evaluation model

260 views • 9 slides
Evaluation of the Implementation of an Abstract Interpretation Algorithm using Tabled CLP n

Evaluation of the Implementation of an Abstract Interpretation Algorithm using Tabled CLP n

ICLP19 Las Cruces, September 22, 2019 Evaluation of the Implementation of an Abstract Interpretation Algorithm using Tabled CLP n Arias 1 , 2 Manuel Carro 1 , 2 Joaqu 1 IMDEA Software Institute, 2 Universidad Polit ecnica de Madrid

468 views • 42 slides
Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review

Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review

10/26/2012 CSE 573: Artificial Intelligence Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review MDPs New MDP Algorithm: Q-value iteration Review Q-learning Large MDPs Linear function

415 views • 8 slides
An Algorithm for Suffix Stripping Evaluation Algorithm Porter (1980) Notations Rules Further

An Algorithm for Suffix Stripping Evaluation Algorithm Porter (1980) Notations Rules Further

Suffix Stripping Sabrina Galasso &amp; Eyal Schejter Introduction Information Retrieval Suffix Stripping An Algorithm for Suffix Stripping Evaluation Algorithm Porter (1980) Notations Rules Further Issues References Sabrina Galasso

757 views • 47 slides
Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning BUPT Pengda Qin ,

Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning BUPT Pengda Qin ,

Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning BUPT Pengda Qin , Weiran Xu and William Wang 1 Outline Motivation Algorithm Experiments Conclusion 2 Outline Motivation Algorithm

406 views • 28 slides

More recommend