ensemble methods
play

Ensemble methods CS 446 Why ensembles? Standard machine learning - PowerPoint PPT Presentation

Oct 31, 2023 •37 likes •967 views

Ensemble methods CS 446 Why ensembles? Standard machine learning setup: We have some data. We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). We output the best on a validation set. 1 / 27 Why ensembles? Standard

  1. Ensemble methods CS 446

  2. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. 1 / 27

  3. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. Question: can we do better than the best? 1 / 27

  4. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. Question: can we do better than the best? What if we use an ensemble/aggregate/combination ? 1 / 27

  5. Why ensembles? Standard machine learning setup: ◮ We have some data. ◮ We train 10 predictors (3-nn, least squares, SVM, ResNet, . . . ). ◮ We output the best on a validation set. Question: can we do better than the best? What if we use an ensemble/aggregate/combination ? We’ll consider two approaches: boosting and bagging . 1 / 27

  6. Bagging 2 / 27

  7. Bagging? This first approach is based upon a simple idea: ◮ If the predictors have indepedent errors , a majority vote of their outputs should be good. Let’s first check this. 3 / 27

  8. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). 4 / 27

  9. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . 4 / 27

  10. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 10 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  11. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 20 0.175 0.150 0.125 0.100 0.075 0.050 0.025 0.000 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  12. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 30 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  13. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 40 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  14. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 50 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  15. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . #classifiers = n = 60 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  16. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 2, fraction red = 0.16 0.5 0.4 0.3 0.2 0.1 0.0 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 4 / 27

  17. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 3, fraction red = 0.064 0.4 0.3 0.2 0.1 0.0 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  18. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 4, fraction red = 0.0256 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  19. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 5, fraction red = 0.01024 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  20. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 6, fraction red = 0.004096 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  21. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Red: all classifiers wrong. #classifiers = n = 7, fraction red = 0.0016384 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  22. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 10, fraction green = 0.366897 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  23. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 20, fraction green = 0.244663 0.175 0.150 0.125 0.100 0.075 0.050 0.025 0.000 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  24. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 30, fraction green = 0.175369 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  25. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 40, fraction green = 0.129766 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  26. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 50, fraction green = 0.0978074 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

  27. Combining classifiers Suppose we have n classifiers. Suppose each is wrong independently with probability 0 . 4 . Model classifier errors as random variables ( Z i ) n i =1 (thus E ( Z i ) = 0 . 4 ). We can model the distribution of errors with Binom ( n, 0 . 4) . Green: at least half classifiers wrong. #classifiers = n = 60, fraction green = 0.0746237 0.10 0.08 0.06 0.04 0.02 0.00 0.0 0.2 0.4 0.6 0.8 1.0 4 / 27

Recommend

Introduction to ensemble methods EN S EMBLE METH ODS IN P YTH ON Romn de las Heras Data

Introduction to ensemble methods EN S EMBLE METH ODS IN P YTH ON Romn de las Heras Data

Introduction to ensemble methods EN S EMBLE METH ODS IN P YTH ON Romn de las Heras Data Scientist, SAP / Agile Solutions Choosing the best model ENSEMBLE METHODS IN PYTHON Surveys ENSEMBLE METHODS IN PYTHON Prerequisite knowledge

582 views • 18 slides
Ensemble Methods Albert Bifet May 2012 COMP423A/COMP523A Data Stream Mining Outline 1.

Ensemble Methods Albert Bifet May 2012 COMP423A/COMP523A Data Stream Mining Outline 1.

Ensemble Methods Albert Bifet May 2012 COMP423A/COMP523A Data Stream Mining Outline 1. Introduction 2. Stream Algorithmics 3. Concept drift 4. Evaluation 5. Classification 6. Ensemble Methods 7. Regression 8. Clustering 9. Frequent

696 views • 23 slides
Decision trees and Ensemble methods Camilo Fosco CS109A Introduction to Data Science Pavlos

Decision trees and Ensemble methods Camilo Fosco CS109A Introduction to Data Science Pavlos

Advanced Section #7: Decision trees and Ensemble methods Camilo Fosco CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader 1 Outline Decision trees Metrics Tree-building algorithms Ensemble methods

710 views • 58 slides
CSC 411: Lecture 17: Ensemble Methods I Class based on Raquel Urtasun & Rich Zemels

CSC 411: Lecture 17: Ensemble Methods I Class based on Raquel Urtasun & Rich Zemels

CSC 411: Lecture 17: Ensemble Methods I Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto March 23, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 17-Ensemble Methods I March 23, 2016 1 / 34 Today

751 views • 34 slides
What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015

What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015

What is it? Instrument or ensemble? Lars Bo Andersen, Humans and IT research seminar, 13/5-2015 Instrument <> Ensemble Our instruments Concepts and methods Empirical stuff Ensembles From French ensemble "all the parts of a thing

285 views • 12 slides
Overview of Decision Trees, Ensemble Methods and Reinforcement Learning CMSC 678 UMBC Outline

Overview of Decision Trees, Ensemble Methods and Reinforcement Learning CMSC 678 UMBC Outline

Overview of Decision Trees, Ensemble Methods and Reinforcement Learning CMSC 678 UMBC Outline Decision Trees Ensemble Methods Bagging Random Forests Reinforcement Learning Decision Trees 20 Questions: http://20q.net/ Goals: 1.

771 views • 62 slides
Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable Selection Variable Selection Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection Variable selection Variable Selection Bias in Classification bias bias rpart Trees and Ensemble Methods

183 views • 6 slides
Chapter 6. Ensemble Methods Wei Pan Division of Biostatistics, School of Public Health,

Chapter 6. Ensemble Methods Wei Pan Division of Biostatistics, School of Public Health,

Chapter 6. Ensemble Methods Wei Pan Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, MN 55455 Email: weip@biostat.umn.edu PubH 7475/8475 Wei Pan c Introduction Have a base learner/algorithm;

497 views • 32 slides
Ensemble Methods Roman Kern KDDM2 Roman Kern, ISDS, TU Graz 1 > www.tugraz.at KDDM2 >

Ensemble Methods Roman Kern KDDM2 Roman Kern, ISDS, TU Graz 1 > www.tugraz.at KDDM2 >

www.tugraz.at > Motivation : Consider Kaggle, routinely the winners employ Ensemble Methods ensembles to gain an advantage. > Goal : In this lecture, the main approaches for ensembles will SCIENCE be presented and their main assumptions.

2.16k views • 12 slides
CS570 Data Mining Classification: Ensemble Methods Cengiz Gnay Dept. Math & CS, Emory

CS570 Data Mining Classification: Ensemble Methods Cengiz Gnay Dept. Math & CS, Emory

CS570 Data Mining Classification: Ensemble Methods Cengiz Gnay Dept. Math & CS, Emory University Fall 2013 Some slides courtesy of Han-Kamber-Pei, Tan et al. , and Li Xiong Gnay (Emory) Classification: Ensemble Methods Fall 2013 1 /

580 views • 23 slides
Ensemble Methods Yingyu Liang Computer Sciences 760 Fall 2017

Ensemble Methods Yingyu Liang Computer Sciences 760 Fall 2017

Ensemble Methods Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven, David Page, Jude Shavlik, Tom

701 views • 23 slides
Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble

Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble

Boosting (ensemble) Module 4 - Ensemble classifiers - Objectives module 4: boosting (ensemble models) LEARNING PERFORMANCE REPRESENTATION DATA PROBLEM RAW DATA CLUSTERING EVALUATION FEATURES UCI datasets unigrams 20newsgroups SUPERVISED

926 views • 34 slides
A comparison of two ensemble generation methods based on oceanic singular vectors and atmospheric

A comparison of two ensemble generation methods based on oceanic singular vectors and atmospheric

A comparison of two ensemble generation methods based on oceanic singular vectors and atmospheric lagged initialization for decadal predictions MiKlip Project Module A (WP: A Coordination) Camille Marini, Yulia Polkova, Armin K hl and

601 views • 13 slides
Derivative-Free Methods for Machine Learning Tasks Inverse Problem Formulations The Ensemble

Derivative-Free Methods for Machine Learning Tasks Inverse Problem Formulations The Ensemble

EnKF Kovachki & Stuart Derivative-Free Methods for Machine Learning Tasks Inverse Problem Formulations The Ensemble Kalman Filter Ensemble Kalman Filter Numerics Nikola B. Kovachki 1 Andrew M. Stuart 1 1 Computational and Mathematical

353 views • 32 slides
Using Bayesian Model Probability with ensemble methods to quantify uncertainty in reservoir

Using Bayesian Model Probability with ensemble methods to quantify uncertainty in reservoir

Using Bayesian Model Probability with ensemble methods to quantify uncertainty in reservoir modelling with multiple prior scenarios Sigurd Ivar Aanonsen, NORCE Energy Svenn Tveit, NORCE Energy Mathias Alerini, Equinor ASA EnKF Workshop, Voss,

473 views • 43 slides
Earthquake Forecasting Ensemble Methods for Merging Models Alexander K. Christensen Dr.

Earthquake Forecasting Ensemble Methods for Merging Models Alexander K. Christensen Dr.

Earthquake Forecasting Ensemble Methods for Merging Models Alexander K. Christensen Dr. Maximilian J. Werner Motivation What is forecasting? What can we actually forecast? What can we do with this? We developed a new

389 views • 18 slides
Prediction in kernelized output spaces: output kernel trees and ensemble methods Pierre Geurts

Prediction in kernelized output spaces: output kernel trees and ensemble methods Pierre Geurts

Prediction in kernelized output spaces: output kernel trees and ensemble methods Pierre Geurts Florence dAlchBuc IBISC CNRS, Universit dEvry, GENOPOLE, Evry, France Department of EE and CS, University of Lige, Belgium 25 janvier

631 views • 52 slides
Ensemble Methods + Recommender Systems Matt Gormley Lecture 21 Nov. 4, 2019 1 Reminders

Ensemble Methods + Recommender Systems Matt Gormley Lecture 21 Nov. 4, 2019 1 Reminders

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Midterm Review + Ensemble Methods + Recommender Systems Matt Gormley Lecture 21 Nov. 4, 2019 1 Reminders

967 views • 79 slides
Data fusion based gene function prediction using ensemble methods Matteo Re and Giorgio Valentini

Data fusion based gene function prediction using ensemble methods Matteo Re and Giorgio Valentini

BITS annual meeting Data fusion based gene function prediction using ensemble methods Matteo Re and Giorgio Valentini D.S.I. - Dipartimento di Scienze dellInformazione Universit` a degli Studi di Milano March 19, 2009 Matteo Re and

389 views • 19 slides
Ensemble Methods Or, Model Combination Based on lecture by Parikshit Ram Numerous Possible

Ensemble Methods Or, Model Combination Based on lecture by Parikshit Ram Numerous Possible

CSE 6242/CX 4242 Ensemble Methods Or, Model Combination Based on lecture by Parikshit Ram Numerous Possible Classifiers! Classifier Training Cross Testing Accuracy time validation time kNN None Can be slow Slow ?? classifier

585 views • 24 slides
Solving Complex Machine Learning Problems with Ensemble Methods ECML/PKDD 2013 Workshop .

Solving Complex Machine Learning Problems with Ensemble Methods ECML/PKDD 2013 Workshop .

. Solving Complex Machine Learning Problems with Ensemble Methods ECML/PKDD 2013 Workshop . Ioannis Katakis, Daniel Hernndez-Lobato, Gonzalo Martnez-Muoz and Ioannis Partalas National and Kapodistrian University of Athens Universidad

388 views • 18 slides
Model Building: Ensemble Methods Max Kuhn and Kjell Johnson Nonclinical Statistics, Pfizer 1 1

Model Building: Ensemble Methods Max Kuhn and Kjell Johnson Nonclinical Statistics, Pfizer 1 1

Model Building: Ensemble Methods Max Kuhn and Kjell Johnson Nonclinical Statistics, Pfizer 1 1 Splitting Example Boston Housing Searching though the first left split ( ), the best split again uses the lower status % In the

646 views • 40 slides
Ensemble Methods (Model Combination) Duen Horng (Polo) Chau Assistant Professor

Ensemble Methods (Model Combination) Duen Horng (Polo) Chau Assistant Professor

http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Ensemble Methods (Model Combination) Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech Parishit Ram

506 views • 11 slides
Ensemble Methods + Recommender Systems Matt Gormley Lecture 28 Apr. 29, 2019 1 Reminders

Ensemble Methods + Recommender Systems Matt Gormley Lecture 28 Apr. 29, 2019 1 Reminders

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Ensemble Methods + Recommender Systems Matt Gormley Lecture 28 Apr. 29, 2019 1 Reminders Homework 9: Learning

888 views • 61 slides

More recommend