measuring model performance or error
play

Measuring model performance or error Introduction to Machine - PowerPoint PPT Presentation

Mar 08, 2024 •530 likes •1.31k views

INTRODUCTION TO MACHINE LEARNING Measuring model performance or error Introduction to Machine Learning Is our model any good? Context of task Accuracy Computation time Interpretability 3 types of tasks

  1. INTRODUCTION TO MACHINE LEARNING Measuring model performance or error

  2. Introduction to Machine Learning Is our model any good? ● Context of task ● Accuracy ● Computation time ● Interpretability ● 3 types of tasks ● Classification ● Regression ● Clustering

  3. Introduction to Machine Learning Classification ● Accuracy and Error ● System is right or wrong ● Accuracy goes up when Error goes down correctly classified instances Accuracy = total amount of classified instances Error = 1 - Accuracy

  4. Introduction to Machine Learning Example ● Squares with 2 features: small/big and solid/do � ed ● Label: colored/not colored ● Binary classification problem

  5. Introduction to Machine Learning Example Truth Predicted ✔ ✘ ✘ ✔ ✔ ✔ ✔ ✔ 3 = = 60% 5 ✔ ✔ ✔ ✘ ✘

  6. Introduction to Machine Learning Example Truth Predicted ✔ ✘ ✘ ✔ ✔ ✔ ✔ ✔ 3 = = 60% 5 ✔ ✔ ✔ ✘ ✘

  7. Introduction to Machine Learning Limits of accuracy ● Classifying very rare heart disease ● Classify all as negative (not sick) ● Predict 99 correct (not sick) and miss 1 ● Accuracy: 99% ● Bogus… you miss every positive case!

  8. Introduction to Machine Learning Confusion matrix ● Rows and columns contain all available labels ● Each cell contains frequency of instances that are classified in a certain way

  9. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction P N p TP FN Truth n FP TN

  10. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction True Positives P N Prediction: P Truth: P p TP FN Truth n FP TN

  11. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction True Negatives P N Prediction: N Truth: N p TP FN Truth n FP TN

  12. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction False Negatives P N Prediction: N Truth: P p TP FN Truth n FP TN

  13. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction False Positives P N Prediction: P Truth: N p TP FN Truth n FP TN

  14. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction P N p TP FN Truth n FP TN

  15. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Precision P N TP/(TP+FP) p TP FN Truth n FP TN

  16. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Precision P N TP/(TP+FP) p TP FN Truth n FP TN

  17. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Recall P N TP/(TP+FN) p TP FN Truth n FP TN

  18. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Recall P N TP/(TP+FN) p TP FN Truth n FP TN

  19. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2

  20. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔

  21. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔ ✔

  22. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘

  23. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘

  24. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔ ✔ ✔

  25. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔ ✘ ✘ ✔ ✔

  26. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘ ✔

  27. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘ ✔

  28. Introduction to Machine Learning Rare heart disease ● Accuracy: 99/(99+1) = 99% ● Recall: 0/1 = 0% ● Precision: undefined — no positive predictions Prediction P N p 0 1 Truth n 0 99

  29. Introduction to Machine Learning Regression: RMSE ● Root Mean Squared Error (RMSE) ● Mean distance between estimates and regression line ● 12 ● ● ● ● ● ● ● 11 ● ● ● ● ● ● ● ● ● 10 ● ● X2 ● ● ● 9 ● ● ● ● ● ● 8 ● ● ● ● ● ● ● ● ● 7 ● ● ● 6 6 7 8 9 10 11 12 X1

  30. Introduction to Machine Learning Clustering ● No label information ● Need distance metric between points

  31. Introduction to Machine Learning Clustering ● Performance measure consists of 2 elements ● Similarity within each cluster ● Similarity between clusters 


  32. Introduction to Machine Learning Within cluster similarity ● Within sum of squares (WSS) ● Diameter 10 ● ● ● ● Minimize ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● X2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 5 − 5 0 5 10 X1

  33. Introduction to Machine Learning Between cluster similarity ● Between cluster sum of squares (BSS) ● Intercluster distance 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Maximize ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● X2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 5 − 5 0 5 10 X1

Recommend

Introduction Questions: Assessment of the expected error of a learning algorithm: Is the error

Introduction Questions: Assessment of the expected error of a learning algorithm: Is the error

Introduction Training Measuring Classifier Performance Comparing Classifiers Introduction Training Measuring Classifier Performance Comparing Classifiers Introduction Questions: Assessment of the expected error of a learning algorithm: Is

291 views • 5 slides
Synchronization in duet performance: Testing the two-person phase error correction model Dirk

Synchronization in duet performance: Testing the two-person phase error correction model Dirk

Synchronization in duet performance: Testing the two-person phase error correction model Dirk Vorberg Institut fr Psychologie Technische Universitt Braunschweig Braunschweig, Germany RPPW2005, Alden Biesen Overview 1. How do ensemble

574 views • 29 slides
Kentucky State Model roup June 20, 2007 Users G The List Model Error Report

Kentucky State Model roup June 20, 2007 Users G The List Model Error Report

Possible User Interface Add-Ons Kentucky State Model roup June 20, 2007 Users G The List Model Error Report System Performance Report Mapping Macros Accessibility Index Report Air Quality Post-Processor Select /

261 views • 14 slides
Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering

Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering

Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering Department Curriculum of Malaysian Polytechnics (The he role o of KPIs Is to assess a and improve the he Ou Outcomes-based E Educati tion)

741 views • 47 slides
Overview ROI: Framework ROI: Logic Model ROI: Model ROI: Framework A performance

Overview ROI: Framework ROI: Logic Model ROI: Model ROI: Framework A performance

Measuring Return On Investment (ROI) for State Government Overview ROI: Framework ROI: Logic Model ROI: Model ROI: Framework A performance measure used to evaluate the efficiency of an investment ROI = Impact from program Impact

180 views • 4 slides
Measurement of Timing Error Detection Performance of Software-based Error Detection Mechanisms

Measurement of Timing Error Detection Performance of Software-based Error Detection Mechanisms

Measurement of Timing Error Detection Performance of Software-based Error Detection Mechanisms and Its Correlation with Simulation Yutaka Masuda, Masanori Hashimoto, Takao Onoye Dept. of Information Systems Eng., Osaka University

376 views • 18 slides
Does Certificate Transparency Break the Web? Measuring Adoption and Error Rate Emily Stark , Ryan

Does Certificate Transparency Break the Web? Measuring Adoption and Error Rate Emily Stark , Ryan

Does Certificate Transparency Break the Web? Measuring Adoption and Error Rate Emily Stark , Ryan Sleevi, Rijad Muminovic, Devon OBrien, Eran Messeri, Adrienne Porter Felt, Brendan McMillion, Parisa Tabriz estark@chromium.org How

484 views • 31 slides
Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance

Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance

Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance Marches On ... But what is performance ? CSE 141, S2'06 Jeff Brown The bottom line: Performance Time to Throughput Car Speed Passengers Bay

274 views • 23 slides
Overfi'ng and Model selec1on Aar$ Singh Machine Learning

Overfi'ng and Model selec1on Aar$ Singh Machine Learning

Overfi'ng and Model selec1on Aar$ Singh Machine Learning 10-701/15-781 Feb 20, 2014 True vs. Empirical Error True Error : Target performance

451 views • 32 slides
Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to

Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to

Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to Larry Carter, UCSD 1 Performance Marches On ... 1200 DEC Alpha 21264/600 1100 1000 900 800 700 Performance 600 500 DEC Alpha 5/500 400

728 views • 29 slides
k -Step Ahead Prediction Error Model 1. k -Step Ahead Prediction Error Model 1. ARMAX model is

k -Step Ahead Prediction Error Model 1. k -Step Ahead Prediction Error Model 1. ARMAX model is

k -Step Ahead Prediction Error Model 1. k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z ) y ( n ) = B ( z ) u ( n k ) + C ( z ) ( n ) k -Step Ahead Prediction Error Model 1. ARMAX model is

1.52k views • 127 slides
How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP

How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP

Agenda ! How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP performance ! ! Best practices ! ! Task parallelism !! !" #" Memory access patterns ! Correctness versus performance ! It

166 views • 14 slides
Error model Independent errors - error probability per bit p Error probability per word

Error model Independent errors - error probability per bit p Error probability per word

379 views • 8 slides
k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z )

k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z )

k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z ) y ( n ) = B ( z ) u ( n k ) + C ( z ) ( n ) u - input y - output - white noise k - delay A , B , C are polynomials in z 1 All

280 views • 16 slides
Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What

Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What

Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What is the best method? Risk adjusted performance measures Multi-factor models Peer group factors Return based factor Asset based

416 views • 24 slides
Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and

Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and

Introduction CPU Peformance and Its Factors Evaluating Performance Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and Its Factors Evaluating Performance Outline 1 Introduction 2 CPU Peformance and

988 views • 65 slides
Measuring Social Performance in LAC Micol Pistelli, Director of Social Performance, MIX December

Measuring Social Performance in LAC Micol Pistelli, Director of Social Performance, MIX December

Microfinance Information Exchange Measuring Social Performance in LAC Micol Pistelli, Director of Social Performance, MIX December 2012 The Premier Source for Microfinance Data and Analysis This presentation is the proprietary and/or

790 views • 6 slides
An Error Model for Multi-threaded Single-node Applications, and Its Implementation Lena

An Error Model for Multi-threaded Single-node Applications, and Its Implementation Lena

An Error Model for Multi-threaded Single-node Applications, and Its Implementation Lena Feinbube, Daniel Richter, and Andreas Polze Operating Systems & Middleware Group Hasso Plattner Institute at University of Potsdam, Germany An Error

696 views • 41 slides
A Two-Factor Error Model for Quantitative Steganalysis Security, Steganography and Watermarking

A Two-Factor Error Model for Quantitative Steganalysis Security, Steganography and Watermarking

A TWO-FACTOR ERROR MODEL FOR QUANTITATIVE STEGANALYSIS A Two-Factor Error Model for Quantitative Steganalysis Security, Steganography and Watermarking of Multimedia Contents SPIE 2006 San Jos California USA Rainer Bhme

574 views • 29 slides
The CPU Performance Equation 40 The Performance Equation (PE) We would like to model how

The CPU Performance Equation 40 The Performance Equation (PE) We would like to model how

The CPU Performance Equation 40 The Performance Equation (PE) We would like to model how architecture impacts performance (latency) This means we need to quantify performance in terms of architectural parameters. Instruction Count

383 views • 37 slides
CSCE 478/878 Lecture 5: Evaluating error D ( h ) Pr x D [ f ( x ) = h ( x )]

CSCE 478/878 Lecture 5: Evaluating error D ( h ) Pr x D [ f ( x ) = h ( x )]

Outline Two Definitions of Error Sample error vs. true error The true error of hypothesis h with respect to target function f and distribution D is the probability that h will misclassify an instance drawn at random accord- Confidence

95 views • 5 slides
Measuring Well Being and Performance: Measuring Well-Being and Performance: Purpose, Measures and

Measuring Well Being and Performance: Measuring Well-Being and Performance: Purpose, Measures and

UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE UCL ENERGY INSTITUTE Measuring Well Being and Performance: Measuring Well-Being and Performance: Purpose, Measures and Policy A presentation to the

603 views • 19 slides
Measuring the Impact of Docker on Network I/O Performance

Measuring the Impact of Docker on Network I/O Performance

University of Amsterdam Masters Programme in System and Network Engineering Measuring the Impact of Docker on Network I/O Performance Author: Rohprimardho

363 views • 12 slides
ERROR DETECTON & CORRECTION Error Detection EDC= Error Detection and Correction bits

ERROR DETECTON & CORRECTION Error Detection EDC= Error Detection and Correction bits

ERROR DETECTON & CORRECTION Error Detection EDC= Error Detection and Correction bits (redundancy) D = Data protected by error checking, may include header fields Error detection not 100% reliable! protocol may miss some errors,

320 views • 18 slides

More recommend