Introduction to Machine Learning Evaluation: Measures for Regression - PowerPoint PPT Presentation

Jun 28, 2023 •363 likes •416 views

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the definitions of mean squared error (MSE) and mean absolute error (MAE) Understand the connections of MSE and MAE to L2 and L1 loss Know the

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the definitions of mean squared error (MSE) and mean absolute error (MAE) Understand the connections of MSE and MAE to L2 and L1 loss Know the definitions of R 2 and generalized R 2
MEAN SQUARED ERROR n ( y ( i ) − ˆ y ( i ) ) 2 ∈ [ 0 ; ∞ ) MSE = 1 � → L2 loss. n i = 1 Single observations with a large prediction error heavily influence the MSE , as they enter quadratically. 7 15 6 5 10 ^ , y ) 4 L ( y y 6.65 6.65 1.15 5 3 2 1.15 0 1 −4 −2 0 2 4 0 2 4 Residuals = y − y ^ x Similar measures: sum of squared errors (SSE), root mean squared error (RMSE, brings measurement back to the original scale of the outcome). � c Introduction to Machine Learning – 1 / 4
MEAN ABSOLUTE ERROR n | y ( i ) − ˆ MAE = 1 y ( i ) | ∈ [ 0 ; ∞ ) � → L1 loss. n i = 1 Less influenced by large errors and maybe more intuitive than the MSE. 7 15 6 5 10 ^ , y ) 4 L ( y y 2.58 6.65 1.07 5 3 2.58 2 1.07 1.15 0 1 −4 −2 0 2 4 1 2 3 4 5 Residuals = y − y ^ x Similar measures: median absolute error (for even more robustness). � c Introduction to Machine Learning – 2 / 4
R 2 Well-known measure from statistics. n ( y ( i ) − ˆ y ( i ) ) 2 � = 1 − SSE LinMod R 2 = 1 − i = 1 n SSE Intercept ( y ( i ) − ¯ y ) 2 � i = 1 Usually introduced as fraction of variance explained by the model Simpler: compares SSE of constant model (baseline) with complex model (LM) R 2 = 1: all residuals are 0, we predict perfectly, R 2 = 0: we predict as badly as the constant model If measured on the training data, R 2 ∈ [ 0 ; 1 ] (LM must be at least as good as the constant) On other data R 2 can even be negative as there is no guarantee that the LM generalizes better than a constant (overfitting) � c Introduction to Machine Learning – 3 / 4
GENERALIZED R 2 FOR ML A simple generalization of R 2 for ML seems to be: 1 − Loss ComplexModel Loss SimplerModel Works for arbitrary measures (not only SSE), for arbitrary models, on any data set of interest E.g. model vs constant, LM vs non-linear model, tree vs forest, model without some features vs model with them included Fairly unknown; our terminology (generalized R 2 ) is non-standard � c Introduction to Machine Learning – 4 / 4

Recommend

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC Measures

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC Measures compstat-lmu.github.io/lecture_i2ml IMBALANCED BINARY LABELS Classify all as no disease (green) high accuracy. Accuracy Paradox c

451 views • 10 slides

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals Know the definitions of misclassification error rate (MCE) and accuracy (ACC) Understand the entries of a confusion matrix Understand the idea of

704 views • 10 slides

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC visualization compstat-lmu.github.io/lecture_i2ml LABELS: ROC SPACE Plot True Positive Rate and False Positive Rate: 1.00 True Class y C2 +

596 views • 17 slides

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml RESAMPLING Aim: Assess the performance of learning algorithm. Make training sets large (to keep the pessimistic bias small), and reduce variance

778 views • 10 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides

Use and Limitations of Machine Learning in Portfolio Management Overview 1. Brief Introduction

Use and Limitations of Machine Learning in Portfolio Management Overview 1. Brief Introduction to Learning 2. Prediction - Futurecasting - Nowcasting - factor analysis 3. Similarity Measures - recommendation system 4. Generating

307 views • 16 slides

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides

Introduction to Machine Learning k -Nearest Neighbors Regression Learning goals Understand the

Introduction to Machine Learning k -Nearest Neighbors Regression Learning goals Understand the basic idea of k-NN Know different distance measures for different scales of feature variables Understand that k-NN has no optimization step

569 views • 13 slides

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Introduction to Machine Learning What is Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T

220 views • 20 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 18 Outline 1 Classification, Regression, Unsupervised Learning 2 About Dimensionality 3 Drawings and

701 views • 18 slides

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine Learning (las.ethz.ch) What is Machine Learning I: An example Classify email messages as Spam or Non Spam Classical Approach : manual rules

1.27k views • 59 slides

INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is

INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is Machine learning is about predicting the future based on the past. -- Hal Daume III Machine Learning is Machine learning is about predicting

917 views • 59 slides

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study of computational mechanisms that learn from data in order to make predictions and decisions. Example 1: image classification A birdwatcher

932 views • 14 slides

Machine Learning for NLP Data preparation and evaluation Aurlie Herbelot 2019 Centre for

Machine Learning for NLP Data preparation and evaluation Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Introduction 2 Building a statistical NLP application (recap) Choose your data carefully (according to

810 views • 59 slides

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides

Introduction to Machine Learning Evaluation: Training Error compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Training Error compstat-lmu.github.io/lecture_i2ml TRAINING ERROR (also: apparent error / resubstitution error) Learner Dataset D Fit Model Predict Dataset D Train Error c Introduction

293 views • 5 slides

Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different

Machine Learning 1 Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different doctors opinions before undergoing a major surgery Reading user reviews before purchasing a product There are countless

695 views • 22 slides

Introduction to Machine Learning Session 1b: General Introduction Reto West Department of

Introduction to Machine Learning Session 1b: General Introduction Reto West Department of Political Science and International Relations University of Geneva Outline 1 What is Machine Learning? Definition of Machine Learning When Do We Need

374 views • 12 slides

A Gentle Introduction to Machine Learning Supervised learning, unsupervised learning (very

9/23/2020 Outline of Machine Learning Lectures Introduction to machine learning (two lectures) A Gentle Introduction to Machine Learning Supervised learning, unsupervised learning (very brief) First Lecture Reinforcement learning

358 views • 13 slides

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep Learning with Python, Manning, 2017 Machine Learning Flow ( ) ( ) ( ) Data Evaluation Training

783 views • 62 slides

Introduction to Machine Learning Evaluation: Test Error Learning goals training error 0.06

Introduction to Machine Learning Evaluation: Test Error Learning goals training error 0.06 test error Underfitting, Overfitting, High Bias, Low Bias, Understand the definition of test 0.04 Low Variance High Variance MSE error 0.02

394 views • 8 slides

CMSC 678 Introduction to Machine Learning Spring 2019

CMSC 678 Introduction to Machine Learning Spring 2019 https://www.csee.umbc.edu/courses/graduate/678/spring19/ Some slides adapted from Hamed Pirsiavash Outline Welcome! Administrivia Basics of Learning Examples of Machine Learning Frank

1.28k views • 92 slides

A Hands-On Introduction to Automatic Machine Learning Lars Kotthofg University of Wyoming

A Hands-On Introduction to Automatic Machine Learning Lars Kotthofg University of Wyoming larsko@uwyo.edu AutoML Workshop, 28 August 2018, Nanjing 1 Machine Learning Data Machine Learning Predictions 2 Automatic Machine Learning

744 views • 33 slides

Applied Machine Learning Introduction 1 APPLIED MACHINE LEARNING Practicalities Contact

APPLIED MACHINE LEARNING Applied Machine Learning Introduction 1 APPLIED MACHINE LEARNING Practicalities Contact Information of the Instructors APPLIED MACHINE LEARNING Practicalities Slides and exercises will be posted on the website of

1.46k views • 61 slides