LEARNING Outline Confusion Matrix F1 Score Gain and Lift Charts - PowerPoint PPT Presentation

Feb 20, 2024 •238 likes •357 views

Measuring Performance CSCI 447/547 MACHINE LEARNING Outline Confusion Matrix F1 Score Gain and Lift Charts Kolmogorov Smirnov Chart ROC / AUC Regression Metrics Kappa Statistic Confusion Matrix Confusion Matrix

Measuring Performance CSCI 447/547 MACHINE LEARNING
Outline  Confusion Matrix  F1 Score  Gain and Lift Charts  Kolmogorov Smirnov Chart  ROC / AUC  Regression Metrics  Kappa Statistic
Confusion Matrix Confusion Matrix Actual Positive Negative Predict Positive a b Precision a/(a+b) Negative Predictive Negative c d d/(d+c) Value Sensitivity / Recall Specificity Accuracy = (a+d)/(a+b+c+d) a/(a+c) d/(d+b) Confusion Matrix Actual 1 0 Predict 1 3,384 639 Precision 85.7% Negative Predictive 0 16 951 98.3% Value Sensitivity / Recall Specificity Accuracy = 88% 99.6% 59.8%
F1 Score  Good F1 score means you have low false positives and low false negatives  F1 = 2*(Precision * Recall)/(Precision + Recall)  Ranges from 0 to 1  Higher values are better
Gain and Lift Charts  Calculate probability for each observation  Sort in descending order  Split into 10 partitions (deciles)  Calculate correct predictions for each partition
Kolomogorov Smirnov Chart  Measure of the degree of separation between positive and negative distributions
ROC / AUC Curves  Advantage over lift charts is that ROC is (almost) independent of the (possibly fluctuating) accuracy rate  Measures model’s ability to discriminate between positive and negative classes
Regression Metrics  Mean Absolute Error  Gives an idea of magnitude of error but not direction  Mean Squared Error (MSE)  Root Mean Squared Error (RMSE)  Converts MSE back to original magnitude  R 2 (and Adjusted R 2 )  Indication of correlation of predictions to actual values  Range between 0 and 1 with higher being better
Logarithmic Loss (Logloss)  Evaluating the predictions of probabilities of membership in a given class  Smaller is better, 0 is perfect
Kappa Statistic  Cohen’s Kappa  How much better than chance a model is  Range is -1 to 1, with higher being better  Some advise against using this  Dependent on distribution of correct and incorrect predictions, Cohen’s Kappa can be misleading  Power’s Kappa ( Informedness)  Likelihood of making an informed decision over a random guess  Recall + TNR – 1  Range is -1 to 1  1: model is always correct; 0: model is random; -1: model is always incorrect
Summary  Confusion Matrix  F1 Score  Gain and Lift Charts  Kolmogorov Smirnov Chart  ROC / AUC  Regression Metrics  Kappa Statistic

Recommend

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q Federated learning: FedSGD and FedAvg q Related research in federated learning q Open problems Outline q Preliminary: deep learning and SGD q Federated

616 views • 50 slides

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides

Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization

CS6501: T opics in Learning and Game Theory (Fall 2019) Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization Measure Algorithm Performance via Regret Warm-up: A Simple Example 2 Overview of Machine

629 views • 37 slides

Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline

Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Nave Bayes

658 views • 54 slides

Statistical learning Chapter 20, Sections 13 Chapter 20, Sections 13 1 Outline

Statistical learning Chapter 20, Sections 13 Chapter 20, Sections 13 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete data linear

399 views • 13 slides

Multi-Task Active Learning Yi Zhang Outline Active Learning Multi-Task Active Learning

Multi-Task Active Learning Yi Zhang Outline Active Learning Multi-Task Active Learning Linguistic Annotations (ACL 08) Image Classification (CVPR 08) Current Work and Discussions Constraint-Driven Active Learning

945 views • 47 slides

At the Confluence of Logic and Learning Guy Van den Broeck Dagstuhl September 3, 2019 Outline

At the Confluence of Logic and Learning Guy Van den Broeck Dagstuhl September 3, 2019 Outline 1. The AI dilemma: logic vs. learning 2. Deep learning with symbolic knowledge 3. Efficient reasoning during learning 4. New machine learning

1.43k views • 78 slides

Machine Learning CS 486/686: Introduction to Artificial Intelligence 1 Outline Forms of

Machine Learning CS 486/686: Introduction to Artificial Intelligence 1 Outline Forms of Learning Inductive Learning - Decision Trees 2 What is Machine Learning Definition: - A computer program is said to learn from experience E with

754 views • 40 slides

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise, Listwise 10.3. Gathering User Input 10.4. LeToR Evaluation 10.5. Beyond Search Advanced Topics in Information Retrieval / Learning to Rank 2 10.1.

1.01k views • 34 slides

Learning from Observations Chapter 18, Sections 13 Chapter 18, Sections 13 1 Outline

Learning from Observations Chapter 18, Sections 13 Chapter 18, Sections 13 1 Outline Learning agents Inductive learning Decision tree learning Measuring learning performance Chapter 18, Sections 13 2 Learning

653 views • 32 slides

Learning from Observations Chapter 18, Sections 13 Chapter 18, Sections 13 1 Outline

569 views • 30 slides

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline Control learning Control policies that choose optimal actions Q learning Convergence Temporal difference learning 2 Control Learning

576 views • 20 slides

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples RNNs Some slides borrowed from Fei-Fei Li & Justin Johnson &

1.1k views • 55 slides

Outline Wh y Mac hine Learning? What is a w ell-dened learning problem?

Outline Wh y Mac hine Learning? What is a w ell-dened learning problem? An example: learning to pla y c hec k ers What questions should w e ask ab out Mac hine Learning? 1 lecture slides

625 views • 21 slides

Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value

5/7/2012 CSE 473: Artificial Intelligence Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value iteration Q-learning Exploration / exploitation Linear function approximation Linear function

159 views • 12 slides

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression Bayesian Gaussian Mixture Models Non-parametric Bayes 2 Take Away ... 1. Maximum Likelihood Estimate (MLE) = arg max p ( D| )

609 views • 27 slides

Reinforcement learning Chapter 21, Sections 14 Chapter 21, Sections 14 1 Outline

Reinforcement learning Chapter 21, Sections 14 Chapter 21, Sections 14 1 Outline Examples Learning a value function for a fixed policy temporal difference learning Q-learning Function approximation Exploration

562 views • 14 slides

MLCC 2018 Statistical Learning: Basic Concepts Lorenzo Rosasco UNIGE-MIT-IIT Outline Learning

MLCC 2018 Statistical Learning: Basic Concepts Lorenzo Rosasco UNIGE-MIT-IIT Outline Learning from Examples Data Space and Distribution Loss Function and Expected Risk Stability, Overfitting and Regularization MLCC 2017 2 Learning from

801 views • 35 slides

STAT 339 Probabilistic Modeling and Machine Learning 30 January 2017 Colin Reimer Dawson

STAT 339 Probabilistic Modeling and Machine Learning 30 January 2017 Colin Reimer Dawson Outline Data Science and Machine Learning Types of Learning Supervised Learning Unsupervised Learning Discovering Model Complexity Course Outline

239 views • 22 slides

Statistical Learning February 4, 2010 CS 489 / 698 University of Waterloo Outline

Statistical Learning February 4, 2010 CS 489 / 698 University of Waterloo Outline Statistical learning Bayesian learning Maximum a posteriori Maximum likelihood Learning from complete Data Reading: R&N Ch

313 views • 20 slides

Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell

Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell & Norvig Sect 21.1-21.3 What is reinforcement learning Temporal-Difference learning Q-learning 2 CS886 Lecture Slides (c) 2010 K.

306 views • 28 slides

Higher Levels of Learning Diana Skrzydlo, Continuing Lecturer Outline Learning Taxonomies

Designing Exams to Test Higher Levels of Learning Diana Skrzydlo, Continuing Lecturer Outline Learning Taxonomies Assessment is Curriculum Higher Level Questions Examples How you can use these ideas Learning Taxonomies

283 views • 17 slides

BBM406 Fundamentals of Machine Learning Lecture 1: Course outline and logistics An overview

Illustration: Tom Gauld BBM406 Fundamentals of Machine Learning Lecture 1: Course outline and logistics An overview of Machine Learning Aykut Erdem // Hacettepe University // Fall 2019 Todays Schedule Course outline and logistics

1.05k views • 79 slides

Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review

10/26/2012 CSE 573: Artificial Intelligence Todays Outline Reinforcement Learning II Dan Weld Review Reinforcement Learning Review MDPs New MDP Algorithm: Q-value iteration Review Q-learning Large MDPs Linear function

415 views • 8 slides