Testing and Error Estimation Machine Learning Prof Hans Georg - PowerPoint PPT Presentation

Testing and Error Estimation Machine Learning Prof Hans Georg Schaathun Høgskolen i Ålesund 5th February 2016 1

Data set Training set Used up in training Test set x 0 = − 1 w 0 Find error probability w x 1 1 w 2 � w i x i y ′ x 2 y w n x n 2

Testing the Neuron Input Features Class label − 1 x 1 y ′ sign x 2 y f ( x ) x n 3

Testing the Neuron Input Features Class label − 1 t x 1 y ′ sign = ? x 2 y f ( x ) x n 3

Testing the Neuron Input Features Class label − 1 t x 1 Error Yes y ′ sign = ? x 2 y f ( x ) Event No x n Correct Event 3

Testing the Neuron Run for each input vector, and count events. Input Features Class label − 1 t x 1 Error Yes y ′ sign = ? x 2 y f ( x ) Event No x n Correct Event 3

Error Rate What is the error probability? 1. error rate: e e + c 2. e error events 3. c correct classifications 4

Error Rate What is the error probability? — Error probability p e — Estimator: ˆ p e (stochastic 1. error rate: e e + c variable) 2. e error events — Estimate: observation r e 3. c correct classifications (error rate) — Confidence interval: (ˆ p l , ˆ p u ) 4

Confusion Matrix Prediction Malign Benign Malign True False Actual Positive Negative Class Benign False True Positive Negative 5

The ROC curve — We can vary w 0 — Receiver operating characteristic (ROC) 1.0 0.8 True positive rate 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 6

The ROC curve — We can vary w 0 — Receiver operating characteristic (ROC) 1.0 0.8 True positive rate 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 6

The ROC curve — We can vary w 0 — Receiver operating characteristic (ROC) 1.0 0.8 True positive rate 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 6

The ROC curve — We can vary w 0 — Receiver operating characteristic (ROC) 1.0 0.8 True positive rate 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 6

The ROC curve — We can vary w 0 — Receiver operating characteristic (ROC) 1.0 0.8 True positive rate 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 6

Summary Testing 1. Use a test set independent of the training set 2. Test set with known class labels 3. Do recall, and compare to known labels Evaluation 1. Statistical analysis of test results 2. How large test set? 7