Introduction to Machine Learning Evaluation: Training Error - PowerPoint PPT Presentation

Oct 08, 2022 •227 likes •293 views

Introduction to Machine Learning Evaluation: Training Error compstat-lmu.github.io/lecture_i2ml TRAINING ERROR (also: apparent error / resubstitution error) Learner Dataset D Fit Model Predict Dataset D Train Error c Introduction

Introduction to Machine Learning Evaluation: Training Error compstat-lmu.github.io/lecture_i2ml
TRAINING ERROR (also: apparent error / resubstitution error) Learner Dataset D Fit Model Predict Dataset D Train Error � c Introduction to Machine Learning – 1 / 4
EXAMPLE: POLYNOMIAL REGRESSION Sample data from sinusoidal function 0 . 5 + 0 . 4 · sin( 2 π x ) + ǫ with measurement error ǫ . 1.00 ● ● ● ● 0.75 ● Train set ● ● 0.50 y ● ● ● ● 0.25 ● True function ● ● 0.00 0.00 0.25 0.50 0.75 1.00 x Assume data generating process unknown. Try to approximate with a d th-degree polynomial: d f ( x | θ ) = θ 0 + θ 1 x + · · · + θ d x d = � θ j x j . j = 0 � c Introduction to Machine Learning – 2 / 4
EXAMPLE: POLYNOMIAL REGRESSION Models of different complexity , i.e., of different orders of the polynomial are fitted. How should we choose d ? degree True function 1 3 9 ● Train set 1.00 ● ● ● ● 0.75 ● ● 0.50 y ● ● ● ● 0.25 ● ● ● 0.00 0.00 0.25 0.50 0.75 1.00 x d=1: MSE = 0.036: Clear underfitting d=3: MSE = 0.003: Pretty OK? d=9: MSE = 0.001: Clear overfitting Simply using the training error seems to be a bad idea. � c Introduction to Machine Learning – 3 / 4
TRAINING ERROR PROBLEMS Unreliable and overly optimistic estimator of future performance. E.g. training error of 1-NN is always zero, as each observation is its own NN during test time. Goodness-of-fit measures like (classical) R 2 , likelihood, AIC, BIC, deviance are all based on the training error. For models of restricted capacity, and given enough data, the training error may provide reliable information. E.g. LM with p = 5 features, 10 6 training points. But: impossible to determine when training error becomes unreliable. � c Introduction to Machine Learning – 4 / 4

Recommend

Introduction to Machine Learning Evaluation: Test Error Learning goals training error 0.06

Introduction to Machine Learning Evaluation: Test Error Learning goals training error 0.06 test error Underfitting, Overfitting, High Bias, Low Bias, Understand the definition of test 0.04 Low Variance High Variance MSE error 0.02

394 views • 8 slides

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the definitions of mean squared error (MSE) and mean absolute error (MAE) Understand the connections of MSE and MAE to L2 and L1 loss Know the

416 views • 5 slides

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals Know the definitions of misclassification error rate (MCE) and accuracy (ACC) Understand the entries of a confusion matrix Understand the idea of

704 views • 10 slides

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml RESAMPLING Aim: Assess the performance of learning algorithm. Make training sets large (to keep the pessimistic bias small), and reduce variance

778 views • 10 slides

BBM406 Fundamentals of Machine Learning Lecture 6: Learning theory Probability Review Aykut

BBM406 Fundamentals of Machine Learning Lecture 6: Learning theory Probability Review Aykut Erdem // Hacettepe University // Fall 2019 Last time Regularization , Cross-Validation error Validation error the data Training error number

1.01k views • 48 slides

Testing and Error Estimation Machine Learning Prof Hans Georg Schaathun Hgskolen i lesund

Testing and Error Estimation Machine Learning Prof Hans Georg Schaathun Hgskolen i lesund 5th February 2016 1 Data set Training set Used up in training Test set x 0 = 1 w 0 Find error probability w x 1 1 w 2 w i x i y x 2

734 views • 15 slides

Measuring model performance or error Introduction to Machine Learning Is our model any good?

INTRODUCTION TO MACHINE LEARNING Measuring model performance or error Introduction to Machine Learning Is our model any good? Context of task Accuracy Computation time Interpretability 3 types of tasks

1.31k views • 77 slides

Introduction to Multi-Armed Bandits and Reinforcement Learning Training School on Machine

Introduction to Multi-Armed Bandits and Reinforcement Learning Training School on Machine Learning for Communications Paris, 23-25 September 2019 Who am I ? . Hi, Im Lilian Besson finishing my PhD in telecommunication and machine

1.73k views • 127 slides

Machine Learning for NLP SVMs for semantic error detection Aurlie Herbelot 2018 Centre for

Machine Learning for NLP SVMs for semantic error detection Aurlie Herbelot 2018 Centre for Mind/Brain Sciences University of Trento 1 Error Detection and Correction: introduction 2 Error Detection and Correction (EDC) The aim of EDC

567 views • 52 slides

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep Learning with Python, Manning, 2017 Machine Learning Flow ( ) ( ) ( ) Data Evaluation Training

783 views • 62 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Training Neural Nets 1 / 40 Outline 1 The Softmax Simplex 2 Loss and Risk 3 Back-Propagation 4 Stochastic Gradient Descent 5 Regularization 6 Network Depth

675 views • 40 slides

Evaluation of training Learning objectives Design a possible evaluation procedure for

INF3280, 15 Feb 2017 Evaluation of training Learning objectives Design a possible evaluation procedure for Assignment 5 Core literature Chapter 9 Auxiliary literature Holton III (1996) The flawed four-level evaluation model

260 views • 9 slides

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC Measures

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC Measures compstat-lmu.github.io/lecture_i2ml IMBALANCED BINARY LABELS Classify all as no disease (green) high accuracy. Accuracy Paradox c

451 views • 10 slides

Introduction Questions: Assessment of the expected error of a learning algorithm: Is the error

Introduction Training Measuring Classifier Performance Comparing Classifiers Introduction Training Measuring Classifier Performance Comparing Classifiers Introduction Questions: Assessment of the expected error of a learning algorithm: Is

291 views • 5 slides

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC

Introduction to Machine Learning Evaluation: Measures for Binary Classification: ROC visualization compstat-lmu.github.io/lecture_i2ml LABELS: ROC SPACE Plot True Positive Rate and False Positive Rate: 1.00 True Class y C2 +

596 views • 17 slides

Runtime Error Analysis - A Machine Learning Perspective Praful Mangalath University of Colorado,

Runtime Error Analysis - A Machine Learning Perspective Praful Mangalath University of Colorado, Boulder Center for Computational Language and EducAtion Research (CLEAR) April 29th, 2009 Project Summary Runtime Error Analysis 5535

323 views • 18 slides

Evaluation of training Learning objectives Design a possible evaluation procedure for

INF3280, 25 March 2020 Evaluation of training Learning objectives Design a possible evaluation procedure for Assignment 3 Core literature Chapter 9 Auxiliary literature Holton III (1996) The flawed four-level

212 views • 10 slides

An Analysis of Graph Cut Size for Transductive Learning Steve Hanneke Machine Learning

An Analysis of Graph Cut Size for Transductive Learning Steve Hanneke Machine Learning Department Carnegie Mellon University 1 Outline Transductive Learning with Graphs Error Bounds for Transductive Learning Error Bounds Based

691 views • 25 slides

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Introduction to Machine Learning What is Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T

220 views • 20 slides

Introduction to Machine Learning Linear Regression Prof. Andreas Krause Learning and Adaptive

Introduction to Machine Learning Linear Regression Prof. Andreas Krause Learning and Adaptive Systems (las.ethz.ch) Basic Supervised Learning Pipeline Training Data Test Data spam ? Learning Predic- ham Model method ?

566 views • 23 slides

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 18 Outline 1 Classification, Regression, Unsupervised Learning 2 About Dimensionality 3 Drawings and

701 views • 18 slides

Classifiers: Support Vector Machine 1 MACHINE LEARNING What is Classification? Female Adult

MACHINE LEARNING MACHINE LEARNING Classifiers: Support Vector Machine 1 MACHINE LEARNING What is Classification? Female Adult Children Detecting facial attributes He & Zhang, Pattern Recognition, 2011 Sony (Make Believe) Training set

914 views • 80 slides