intelligible models for classification and regression
play

Intelligible Models for Classification and Regression Yin Lou 1 Rich - PowerPoint PPT Presentation

Jun 20, 2023 •36 likes •436 views

Intelligible Models for Classification and Regression Yin Lou 1 Rich Caruana 2 Johannes Gehrke 1 Department of Computer Science 1 Microsoft Research 2 Cornell University Microsoft Corporation Aug. 13, 2012 Yin Lou (Cornell University)

  1. Intelligible Models for Classification and Regression Yin Lou 1 Rich Caruana 2 Johannes Gehrke 1 Department of Computer Science 1 Microsoft Research 2 Cornell University Microsoft Corporation Aug. 13, 2012 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 1 / 31

  2. Motivation Simple Model Linear regression, logistic regression Regression: y = β 0 + β 1 x 1 + ... + β n x n Classification: logit ( y ) = β 0 + β 1 x 1 + ... + β n x n Linear Regression Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 2 / 31

  3. Motivation Simple Model Linear regression, logistic regression Regression: y = β 0 + β 1 x 1 + ... + β n x n Classification: logit ( y ) = β 0 + β 1 x 1 + ... + β n x n Linear Regression Intelligible but usually less accurate Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 2 / 31

  4. Motivation Complex Model Random forest, SVMs with RBF kernel, etc. y = f ( x 1 , ..., x n ) Random Forest Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 3 / 31

  5. Motivation Complex Model Random forest, SVMs with RBF kernel, etc. y = f ( x 1 , ..., x n ) Random Forest Unintelligible but usually more accurate Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 3 / 31

  6. Motivation The tradeoff SVMs with RBF Kernel Random Forest Complexity ? Linear Regression Logistic Regression Intelligibility Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 4 / 31

  7. Motivation The tradeoff SVMs with RBF Kernel Random Forest Complexity ? Linear Regression Logistic Regression Intelligibility Intelligibility is important Medical applications Domains where we want scientific understanding Efficient model engineering Impact of features in a ranker Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 4 / 31

  8. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 5 / 31

  9. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 6 / 31

  10. Generalized Additive Models Developed by Hastie and Tibshirani Regression: y = f 1 ( x 1 ) + ... + f n ( x n ) Classification: logit ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Each feature is “shaped” by shape function f i Intelligible and accurate T. Hastie and R. Tibshirani. Generalized additive models . Chapman & Hall/CRC, 1990. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 7 / 31

  11. Example 2 + √ x 3 + log x 4 + e x 5 + 2 sin x 6 + ǫ y = x 1 + x 2 2 2 2 1 1 1 0 0 0 −1 −1 −1 −2 −2 −2 0 1 2 3 4 0.0 0.5 1.0 1.5 2.0 0 5 10 15 f 1 ( x 1 ) f 2 ( x 2 ) f 3 ( x 3 ) 2 2 2 1 1 1 0 0 0 −1 −1 −1 −2 −2 −2 0 10 20 30 40 50 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 0 1 2 3 4 5 6 f 4 ( x 4 ) f 5 ( x 5 ) f 6 ( x 6 ) Figure: Shape Functions for Synthetic Dataset. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 8 / 31

  12. Model Space Model Form Intelligibility Accuracy Linear Model y = β 0 + β 1 x 1 + ... + β n x n +++ + Generalized Linear Model g ( y ) = β 0 + β 1 x 1 + ... + β n x n +++ + Additive Model y = f 1 ( x 1 ) + ... + f n ( x n ) ++ ++ Generalized Additive Model g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) ++ ++ Full Complexity Model y = f ( x 1 , ..., x n ) + +++ Table: From Linear to Additive Models. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 9 / 31

  13. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 10 / 31

  14. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Functions Splines (SP) Single Tree (TR) Bagged Trees (bagTR) Boosted Trees (bstTR) Boosted Bagged Trees (bbTR) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 11 / 31

  15. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Functions Splines (SP) Single Tree (TR) Bagged Trees (bagTR) Boosted Trees (bstTR) Boosted Bagged Trees (bbTR) Learning Methods Penalized Least Squares (P-LS/P-IRLS) Backfitting (BF) Gradient Boosting (BST) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 11 / 31

  16. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Splines (SP) f i ( x i ) = � d k =1 β k b k ( x i ) 40 30 20 10 0 −10 −20 100 200 300 400 500 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 12 / 31

  17. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Single Tree (TR) f i ( x i ) = RegressionTree ( x i , response ) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 13 / 31

  18. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Bagged Trees (bagTR) f i ( x i ) = 1 � B j =1 RegressionTree ( x i , bootstrap sample j ) B 1 ( + ... + ) B Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 14 / 31

  19. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Boosted Trees (bstTR) f i ( x i ) = � B j =1 RegressionTree ( x i , residual j ) + ... + Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 15 / 31

  20. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Shape Function: Boosted Bagged Trees (bbTR) f i ( x i ) = � B j =1 BaggedRegressionTree ( x i , residual j ) Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 16 / 31

  21. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Learning Method: Penalized Least Squares (P-LS/P-IRLS) Works only on Splines ( f i ( x i ) = � d k =1 β k b k ( x i )) Converts the optimization problem to fitting linear regression/logistic regression with different basis S. Wood. Generalized additive models: an introduction with R . CRC Press, 2006. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 17 / 31

  22. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Learning Method: Backfitting (BF) 1: f j ← 0 2: for m = 1 to M do for j = 1 to n do 3: k � = j f k } N R ← { x ij , y i − � 4: 1 Learn shaping function S : x j → y using R as training dataset 5: f j ← S 6: end for 7: 8: end for T. Hastie and R. Tibshirani. Generalized additive models . Chapman & Hall/CRC, 1990. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 18 / 31

  23. Fitting GAMs g ( y ) = f 1 ( x 1 ) + ... + f n ( x n ) Learning Method: Gradient Boosting (BST) 1: f j ← 0 2: for m = 1 to M do for j = 1 to n do 3: k f k } N R ← { x ij , y i − � 4: 1 Learn shaping function S : x j → y using R as training dataset 5: f j ← f j + S 6: end for 7: 8: end for J. Friedman. Greedy function approximation: a gradient boosting machine. Annals of Statistics , 29:1189–1232, 2001. Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 19 / 31

  24. Contributions First large-scale study that uses trees as shape function for GAMs Novel methods for using trees as shape functions Largest empirical study of fitting GAMs Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 20 / 31

  25. Outline Motivation 1 Towards More Accurate Models 2 Algorithms 3 Experiments 4 Discussion 5 Conclusion 6 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 21 / 31

  26. Datasets Dataset Size Attributes %Pos Concrete 1030 9 - Regression Wine 4898 12 - Delta 7192 6 - CompAct 8192 22 - Music 50000 90 - Synthetic 10000 6 - Spambase 4601 58 39.40 Classification Insurance 9823 86 5.97 Magic 19020 11 64.84 Letter 20000 17 49.70 Adult 46033 9/43 16.62 Physics 50000 79 49.72 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 22 / 31

  27. Methods Shape Least Gradient Backfitting Function Squares Boosting Splines P-LS/P-IRLS BST-SP BF-SP Single Tree N/A BST-TR x BF-TR Bagged Trees N/A BST-bagTR x BF-bagTR Boosted Trees N/A BST-TR x BF-bstTR x Boosted N/A BST-bagTR x BF-bbTR x Bagged Trees Table: Notation for learning methods and shape functions. 9 different methods 5-fold cross validation for each method Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 23 / 31

  28. Results Model Regression Classification Mean Linear/Logistic P-LS/P-IRLS BST-SP BF-SP BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

  29. Results Model Regression Classification Mean Linear/Logistic 1.68 1.22 1.45 P-LS/P-IRLS BST-SP BF-SP BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

  30. Results Model Regression Classification Mean Linear/Logistic 1.68 1.22 1.45 P-LS/P-IRLS BST-SP BF-SP BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest 0.88 0.80 0.84 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

  31. Results Model Regression Classification Mean Linear/Logistic 1.68 1.22 1.45 P-LS/P-IRLS 1.00 1.00 1.00 BST-SP 1.04 1.00 1.02 BF-SP 1.00 1.00 1.00 BST-bagTR2 BST-bagTR3 BST-bagTR4 BST-bagTRX Random Forest 0.88 0.80 0.84 Yin Lou (Cornell University) Intelligible Models Aug. 13, 2012 24 / 31

Recommend

Regression l For classification the output(s) is nominal l In regression the output is continuous

Regression l For classification the output(s) is nominal l In regression the output is continuous

Regression l For classification the output(s) is nominal l In regression the output is continuous Function Approximation l Many models could be used Simplest is linear regression Fit data with the best hyper-plane which "goes

592 views • 35 slides
Linear models for classification. Perceptron. Logistic regression. Petr Po s k P. Po

Linear models for classification. Perceptron. Logistic regression. Petr Po s k P. Po

CZECH TECHNICAL UNIVERSITY IN PRAGUE Faculty of Electrical Engineering Department of Cybernetics Linear models for classification. Perceptron. Logistic regression. Petr Po s k P. Po s k c 2015 Artificial Intelligence 1

484 views • 17 slides
Classification or Regression? Regression Classification: want to learn a discrete target

Classification or Regression? Regression Classification: want to learn a discrete target

Classification or Regression? Regression Classification: want to learn a discrete target variable Machine Learning and Pattern Recognition Regression: want to learn a continuous target variable Linear regression,

398 views • 6 slides
Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335 Cost function Gradient descent Linear classifiers and decision boundaries Dan Sheldon Classification Example: Hand-Written Digits Input: 20

541 views • 5 slides
10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression another discriminative learning method As optimization Gradient descent As matrix inversion (Ordinary Least Squares) Overfitting

1.52k views • 97 slides
Decision Trees TJ Machine Learning Club Classification vs. Regression Classification

Decision Trees TJ Machine Learning Club Classification vs. Regression Classification

Decision Trees TJ Machine Learning Club Classification vs. Regression Classification Classifying photos of fruits Determining whether tumor is benign or malignant Regression Predicting COVID-19 cases given demographic data

505 views • 26 slides
Multiclass Classification Machine Learning So far: Binary Classification We have seen linear

Multiclass Classification Machine Learning So far: Binary Classification We have seen linear

Multiclass Classification Machine Learning So far: Binary Classification We have seen linear models Learning algorithms for linear models Perceptron, Winnow, Adaboost, SVM We will see more soon: Nave Bayes, Logistic Regression

635 views • 24 slides
Theory for Minimum Norm Interpolation: Regression and Classification in High Dimensions Tengyuan

Theory for Minimum Norm Interpolation: Regression and Classification in High Dimensions Tengyuan

Intro. Min-norm Interpolant Regression Classification Theory for Minimum Norm Interpolation: Regression and Classification in High Dimensions Tengyuan Liang Classification: with Pragya Sur (Harvard) Regression: with Sasha Rakhlin (MIT), Xiyu

800 views • 68 slides
Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models Classifiers with a fixed number of parameters can represent a limited set of functions. Learning a model is about picking a good approximation. Typically the

427 views • 39 slides
Analysis of variance and regression Other types of regression models Other types of regression

Analysis of variance and regression Other types of regression models Other types of regression

Analysis of variance and regression Other types of regression models Other types of regression models Counts: Poisson models Ordinal data: Proportional odds models Survival analysis (censored, time-to-event data): Cox proportional

1.02k views • 58 slides
Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear Regression M. Magdon-Ismail CSCI 4100/6100 recap: Approximation Versus Generalization VC Analysis Bias-Variance Analysis E out E in + ( d

390 views • 23 slides
Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical

Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical

MACHINE LEARNING TOOLBOX Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical (i.e. qualitative) target variable Example: will a loan default? Still a form of supervised learning Use a

315 views • 27 slides
E9 205 Machine Learning for Signal Processing Linear Models for Regression and 23-09-2019

E9 205 Machine Learning for Signal Processing Linear Models for Regression and 23-09-2019

E9 205 Machine Learning for Signal Processing Linear Models for Regression and 23-09-2019 Classification Linear Regression Solution to Maximum Likelihood problem is the least squares solution Pseudo Inverse Based Solution Bishop - PRML

84 views • 7 slides
Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725

Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725

Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Linear Models for Classification Feature function representation Weights Nave

483 views • 25 slides
Introduction to Machine Learning Classification and Regression Trees (CART): Basics

Introduction to Machine Learning Classification and Regression Trees (CART): Basics

Introduction to Machine Learning Classification and Regression Trees (CART): Basics compstat-lmu.github.io/lecture_i2ml TREE MODEL AND PREDICTION Classification and Regression Trees, introduced by Breiman Binary splits are constructed top-down

415 views • 6 slides
Data Mining II Regression Heiko Paulheim Regression Classification covered in Data

Data Mining II Regression Heiko Paulheim Regression Classification covered in Data

Data Mining II Regression Heiko Paulheim Regression Classification covered in Data Mining I predict a label from a finite collection e.g., true/false, low/medium/high, ... Regression predict a numerical value

831 views • 67 slides
Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear

Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear

Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear models, assume that the response, Y , has a normal distribution with constant variance. Unlike linear models, however, the mean function may depend

159 views • 4 slides
Machine Learning - MT 2016 8. Classification: Logistic Regression Varun Kanade University of

Machine Learning - MT 2016 8. Classification: Logistic Regression Varun Kanade University of

Machine Learning - MT 2016 8. Classification: Logistic Regression Varun Kanade University of Oxford November 2, 2016 Logistic Regression Logistic Regression is actually a classification method In its simplest form it is a binary (two classes)

486 views • 14 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Least Squares for Classification K-class

467 views • 24 slides
Lecture 11: Logistic Regression Extensions and Evaluating Classification CS109A Introduction to

Lecture 11: Logistic Regression Extensions and Evaluating Classification CS109A Introduction to

Lecture 11: Logistic Regression Extensions and Evaluating Classification CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Lecture Outline Logistic Regression: a Brief Review Classification Boundaries

422 views • 41 slides
Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC

Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC

Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Module 2: Classification Why not Linear Regression? Binary

1.1k views • 77 slides
Introduction to Machine Learning Classification: Logistic Regression

Introduction to Machine Learning Classification: Logistic Regression

Introduction to Machine Learning Classification: Logistic Regression compstat-lmu.github.io/lecture_i2ml MOTIVATION A discriminant approach for directly modeling the posterior probabilities ( x | ) of the labels is logistic regression . For

484 views • 9 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Linear Regression Solution to Maximum

145 views • 11 slides
Analysis of variance and regression November 27, 2007 Other types of regression models Counts

Analysis of variance and regression November 27, 2007 Other types of regression models Counts

Analysis of variance and regression November 27, 2007 Other types of regression models Counts (Poisson models) Ordinal data proportional odds models model control model interpretation Survival analysis Lene Theil

844 views • 65 slides

More recommend