Introduction to Machine Learning Evaluation: Measures for Binary - PowerPoint PPT Presentation

Nov 03, 2023 •335 likes •451 views

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC Measures compstat-lmu.github.io/lecture_i2ml IMBALANCED BINARY LABELS Classify all as no disease (green) high accuracy. Accuracy Paradox c

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC Measures compstat-lmu.github.io/lecture_i2ml
IMBALANCED BINARY LABELS Classify all as “no disease” (green) → high accuracy. Accuracy Paradox � c Introduction to Machine Learning – 1 / 9
IMBALANCED COSTS Classify incorrectly as “no disease” → very high cost � c Introduction to Machine Learning – 2 / 9
CONFUSION MATRIX True Class y + − + Pred. TP FP ˆ y FN TN − + : “positive” class − : “negative” class n + : number of observations in + n − : number of observations in − � c Introduction to Machine Learning – 3 / 9
LABELS: ROC METRICS From the confusion matrix (binary case), we can calculate "ROC" metrics. True Class y + − TP + Pred. TP FP PPV = TP + FP TN ˆ y FN TN NPV = − FN + TN TP TN Accuracy = TP + TN TPR = TNR = TP + FN FP + TN TOTAL True Positive Rate: How many of the true 1s did we predict as 1? True Negative Rate: How many of the true 0s did we predict as 0? Positive Predictive Value: If we predict 1 how likely is it a true 1? Negative Predictive Value: If we predict 0 how likely is it a true 0? � c Introduction to Machine Learning – 4 / 9
HISTORY ROC ROC = receiver operating characteristics Initially developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battlefields. http://media.iwm.org.uk/iwm/mediaLib//39/media-39665/large.jpg Still has the funny name. � c Introduction to Machine Learning – 5 / 9
LABELS: ROC Example � c Introduction to Machine Learning – 6 / 9
MORE METRICS AND ALTERNATIVE TERMINOLOGY Unfortunately, for many concepts in ROC, 2-3 different terms exist. Clickable version/picture source Interactive diagram � c Introduction to Machine Learning – 7 / 9
LABELS: F 1 -MEASURE A measure that balances two conflicting goals Maximising Positive Predictive Value 1 Maximising True Positive Rate 2 is the harmonic mean of PPV and TPR: PPV · TPR F 1 = 2 PPV + TPR Note: still doesn’t account for the number of true negatives. � c Introduction to Machine Learning – 8 / 9
LABELS: F 1 -MEASURE Tabulated F 1 -Score for different TPR (rows) and PPV (cols) combinations. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0 0.00 0.00 0.00 0.00 0.00 0.2 0 0.20 0.27 0.30 0.32 0.33 0.4 0 0.27 0.40 0.48 0.53 0.57 0.6 0 0.30 0.48 0.60 0.69 0.75 0.8 0 0.32 0.53 0.69 0.80 0.89 1.0 0 0.33 0.57 0.75 0.89 1.00 → Tends more towards the lower of the 2 combined values. TPR = 0 or PPV = 0 ⇒ F 1 of 0 Predicting always "neg": F 1 = 0 Predicting always "pos": F 1 = 2 PPV / ( PPV + 1 ) = 2 n + / ( n + + n ) , which will be rather small, if the size of the positive class n + is small. � c Introduction to Machine Learning – 9 / 9

Recommend

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals Know the definitions of misclassification error rate (MCE) and accuracy (ACC) Understand the entries of a confusion matrix Understand the idea of

704 views • 10 slides

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the definitions of mean squared error (MSE) and mean absolute error (MAE) Understand the connections of MSE and MAE to L2 and L1 loss Know the

416 views • 5 slides

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC visualization compstat-lmu.github.io/lecture_i2ml LABELS: ROC SPACE Plot True Positive Rate and False Positive Rate: 1.00 True Class y C2 +

596 views • 17 slides

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml RESAMPLING Aim: Assess the performance of learning algorithm. Make training sets large (to keep the pessimistic bias small), and reduce variance

778 views • 10 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides

Use and Limitations of Machine Learning in Portfolio Management Overview 1. Brief Introduction

Use and Limitations of Machine Learning in Portfolio Management Overview 1. Brief Introduction to Learning 2. Prediction - Futurecasting - Nowcasting - factor analysis 3. Similarity Measures - recommendation system 4. Generating

307 views • 16 slides

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides

Introduction to Machine Learning k -Nearest Neighbors Regression Learning goals Understand the

Introduction to Machine Learning k -Nearest Neighbors Regression Learning goals Understand the basic idea of k-NN Know different distance measures for different scales of feature variables Understand that k-NN has no optimization step

569 views • 13 slides

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Introduction to Machine Learning What is Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T

220 views • 20 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 18 Outline 1 Classification, Regression, Unsupervised Learning 2 About Dimensionality 3 Drawings and

701 views • 18 slides

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine Learning (las.ethz.ch) What is Machine Learning I: An example Classify email messages as Spam or Non Spam Classical Approach : manual rules

1.27k views • 59 slides

INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is

INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is Machine learning is about predicting the future based on the past. -- Hal Daume III Machine Learning is Machine learning is about predicting

917 views • 59 slides

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study of computational mechanisms that learn from data in order to make predictions and decisions. Example 1: image classification A birdwatcher

932 views • 14 slides

Machine Learning for NLP Data preparation and evaluation Aurlie Herbelot 2019 Centre for

Machine Learning for NLP Data preparation and evaluation Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Introduction 2 Building a statistical NLP application (recap) Choose your data carefully (according to

810 views • 59 slides

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides

Introduction to Machine Learning Evaluation: Training Error compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Training Error compstat-lmu.github.io/lecture_i2ml TRAINING ERROR (also: apparent error / resubstitution error) Learner Dataset D Fit Model Predict Dataset D Train Error c Introduction

293 views • 5 slides

Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different

Machine Learning 1 Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different doctors opinions before undergoing a major surgery Reading user reviews before purchasing a product There are countless

695 views • 22 slides

Introduction to Machine Learning Session 1b: General Introduction Reto West Department of

Introduction to Machine Learning Session 1b: General Introduction Reto West Department of Political Science and International Relations University of Geneva Outline 1 What is Machine Learning? Definition of Machine Learning When Do We Need

374 views • 12 slides

A Gentle Introduction to Machine Learning Supervised learning, unsupervised learning (very

9/23/2020 Outline of Machine Learning Lectures Introduction to machine learning (two lectures) A Gentle Introduction to Machine Learning Supervised learning, unsupervised learning (very brief) First Lecture Reinforcement learning

358 views • 13 slides

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep Learning with Python, Manning, 2017 Machine Learning Flow ( ) ( ) ( ) Data Evaluation Training

783 views • 62 slides

Introduction to Machine Learning Evaluation: Test Error Learning goals training error 0.06

Introduction to Machine Learning Evaluation: Test Error Learning goals training error 0.06 test error Underfitting, Overfitting, High Bias, Low Bias, Understand the definition of test 0.04 Low Variance High Variance MSE error 0.02

394 views • 8 slides

CMSC 678 Introduction to Machine Learning Spring 2019

CMSC 678 Introduction to Machine Learning Spring 2019 https://www.csee.umbc.edu/courses/graduate/678/spring19/ Some slides adapted from Hamed Pirsiavash Outline Welcome! Administrivia Basics of Learning Examples of Machine Learning Frank

1.28k views • 92 slides

A Hands-On Introduction to Automatic Machine Learning Lars Kotthofg University of Wyoming

A Hands-On Introduction to Automatic Machine Learning Lars Kotthofg University of Wyoming larsko@uwyo.edu AutoML Workshop, 28 August 2018, Nanjing 1 Machine Learning Data Machine Learning Predictions 2 Automatic Machine Learning

744 views • 33 slides

Applied Machine Learning Introduction 1 APPLIED MACHINE LEARNING Practicalities Contact

APPLIED MACHINE LEARNING Applied Machine Learning Introduction 1 APPLIED MACHINE LEARNING Practicalities Contact Information of the Instructors APPLIED MACHINE LEARNING Practicalities Slides and exercises will be posted on the website of

1.46k views • 61 slides