decision tree learning
play

Decision tree learning Introduction to Machine Learning Task of - PowerPoint PPT Presentation

Oct 14, 2023 •266 likes •1.08k views

INTRODUCTION TO MACHINE LEARNING Decision tree learning Introduction to Machine Learning Task of classification Automatically assign class to observations with features Observation: vector of features , with a class Automatically

  1. INTRODUCTION TO MACHINE LEARNING Decision tree learning

  2. Introduction to Machine Learning Task of classification ● Automatically assign class to observations with features ● Observation: vector of features , with a class ● Automatically assign class to new observation with features , using previous observations ● Binary classification: two classes ● Multiclass classification: more than two classes

  3. Introduction to Machine Learning Example ● A dataset consisting of persons ● Features : age, weight and income ● Class : ● binary : happy or not happy ● multiclass : happy, satisfied or not happy

  4. Introduction to Machine Learning Examples of features ● Features can be numerical ● age: 23, 25, 75, … ● height: 175.3, 179.5, … ● Features can be categorical ● travel_class: first class, business class, coach class ● smokes?: yes, no

  5. Introduction to Machine Learning The decision tree ● Suppose you’re classifying patients as sick or not sick ● Intuitive way of classifying: ask questions Is the patient young or old?

  6. Introduction to Machine Learning The decision tree ● Suppose you’re classifying patients as sick or not sick ● Intuitive way of classifying: ask questions Is the patient young or old? Old

  7. Introduction to Machine Learning The decision tree ● Suppose you’re classifying patients as sick or not sick ● Intuitive way of classifying: ask questions Is the patient young or old? Old Smoked for more than 10 years?

  8. Introduction to Machine Learning The decision tree ● Suppose you’re classifying patients as sick or not sick ● Intuitive way of classifying: ask questions Is the patient young or old? Young Old Vaccinated against the measles? Smoked for more than 10 years?

  9. Introduction to Machine Learning The decision tree ● Suppose you’re classifying patients as sick or not sick ● Intuitive way of classifying: ask questions Is the patient young or old? Young Old Vaccinated against the measles? Smoked for more than 10 years? Yes No Yes No … … … …

  10. Introduction to Machine Learning The decision tree ● Suppose you’re classifying patients as sick or not sick ● Intuitive way of classifying: ask questions Is the patient young or old? Young Old Vaccinated against the measles? Smoked for more than 10 years? Yes No Yes No … … … … It’s a decision tree!!!

  11. Introduction to Machine Learning Define the tree A B C D E F G

  12. Introduction to Machine Learning Define the tree A Nodes B C D E F G

  13. Introduction to Machine Learning Define the tree A Edges B C D E F G

  14. Introduction to Machine Learning Define the tree Root A B C D E F G

  15. Introduction to Machine Learning Define the tree Root A B C D E F G Leafs

  16. Introduction to Machine Learning Define the tree Root A Children of A B C Children of B, C D E F G Grandchildren of A

  17. Introduction to Machine Learning Define the tree Root A Children of A B C Children of B, C D E F G Grandchildren of A Leafs

  18. Introduction to Machine Learning Questions to ask age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick

  19. Introduction to Machine Learning Categorical feature ● Can be a feature test on itself ● travel_class: coach, business or first travel_class coach first business … … …

  20. Introduction to Machine Learning Classifying with the tree Observation: patient of 40 years, vaccinated and didn’t smoke age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick

  21. Introduction to Machine Learning Classifying with the tree Observation: patient of 40 years, vaccinated and didn’t smoke age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick

  22. Introduction to Machine Learning Classifying with the tree Observation: patient of 40 years, vaccinated and didn’t smoke age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick

  23. Introduction to Machine Learning Classifying with the tree Observation: patient of 40 years, vaccinated and didn’t smoke age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick

  24. Introduction to Machine Learning Classifying with the tree Observation: patient of 40 years, vaccinated and didn’t smoke age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick

  25. Introduction to Machine Learning Classifying with the tree Observation: patient of 40 years, vaccinated and didn’t smoke age <= 18 yes no vaccinated smoked yes no yes no not not sick sick sick sick Prediction: not sick

  26. Introduction to Machine Learning Learn a tree ● Use training set ● Come up with queries (feature tests) at each node

  27. Introduction to Machine Learning Split into parts training set 2 parts for binary test age <= 18 yes no part of training set part of training set part of training set part of training set TRUE FALSE

  28. Introduction to Machine Learning part of training set part of training set feature test feature test yes no yes no part of training set part of training set part of training set part of training set part of training set part of training set part of training set part of training set

  29. Introduction to Machine Learning part of training set part of training set part of training set part of training set keep splitting until leafs contain small portion of training set

  30. Introduction to Machine Learning Learn the tree ● Goal : end up with pure leafs — leafs that contain observations of one particular class leaf part of training set class 1 class class 2

  31. Introduction to Machine Learning Learn the tree ● Goal : end up with pure leafs — leafs that contain observations of one particular class leaf leaf ● In practice : almost never the case — noise ● When classifying new instances part of training set part of training set class 1 class 1 class class 2 class 2 ● end up in leaf

  32. Introduction to Machine Learning Learn the tree ● Goal : end up with pure leafs — leafs that contain observations of one particular class leaf ● In practice : almost never the case — noise ● When classifying new instances part of training set class 2 class 1 ● end up in leaf ● assign class of majority of training instances

  33. Introduction to Machine Learning Learn the tree ● At each node ● Iterate over di ff erent feature tests ● Choose the best one ● Comes down to two parts ● Make list of feature tests ● Choose test with best split

  34. Introduction to Machine Learning Construct list of tests ● Categorical features ● Parents/grandparents/… didn’t use the test yet ● Numerical features ● Choose feature ● Choose threshold

  35. Introduction to Machine Learning Choose best feature test ● More complex ● Use spli � ing criteria to decide which test to use ● Information gain ~ entropy

  36. Introduction to Machine Learning Information gain ● Information gained from split based on feature test ● Test leads to nicely divided classes 
 -> high information gain ● Test leads to scrambled classes 
 -> low information gain ● Test with highest information gain will be chosen

  37. Introduction to Machine Learning Pruning ● Number of nodes influences chance on overfit ● Restrict size — higher bias ● Decrease chance on overfit ● Pruning the tree

  38. INTRODUCTION TO MACHINE LEARNING Let’s practice!

  39. INTRODUCTION TO MACHINE LEARNING k-Nearest Neighbors

  40. Introduction to Machine Learning Instance-based learning ● Save training set in memory ● No real model like decision tree ● Compare unseen instances to training set ● Predict using the comparison of unseen data and the training set

  41. Introduction to Machine Learning k-Nearest Neighbor ● Form of instance-based learning ● Simplest form: 1-Nearest Neighbor or Nearest Neighbor

  42. Introduction to Machine Learning Nearest Neighbor - example ● 2 features : X1 and X2 ● Class : red or blue ● Binary classification

  43. Introduction to Machine Learning Nearest Neighbor - example

  44. Introduction to Machine Learning Nearest Neighbor - example ● Save complete training set

  45. Introduction to Machine Learning Nearest Neighbor - example ● Save complete training set ● Given: unseen observation with features X = (1.3, -2)

  46. Introduction to Machine Learning Nearest Neighbor - example ● Save complete training set ● Given: unseen observation with features X = (1.3, -2) ● Compare training set with new observation

  47. Introduction to Machine Learning Nearest Neighbor - example ● Save complete training set ● Given: unseen observation with features X = (1.3, -2) ● Compare training set with new observation ● Find closest observation — nearest neighbor — and assign same class just Euclidean distance, nothing fancy

  48. Introduction to Machine Learning k-Nearest Neighbors ● k is the amount of neighbors ● If k = 5 ● Use 5 most similar observations (neighbors) ● Assigned class will be the most represented class within the 5 neighbors

Recommend

Decision Tree and Automata Learning Stefan Edelkamp 1 Overview - Decision tree representation

Decision Tree and Automata Learning Stefan Edelkamp 1 Overview - Decision tree representation

Decision Tree and Automata Learning Stefan Edelkamp 1 Overview - Decision tree representation - Top Down Induction, Attribute Selection - Entropy, Information gain - ID3 learning algorithm - Overfitting - Continous, multi-valued, costly and

718 views • 33 slides
Decision Tree Decision Trees A decision tree is a decision support tool that uses a tree-like

Decision Tree Decision Trees A decision tree is a decision support tool that uses a tree-like

Decision Tree Decision Trees A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible MSE 2400 EaLiCaRA consequences, including chance event Dr. Tom Way outcomes, resource costs, and

568 views • 8 slides
Decision Tree Learning: Part 2 CS 760@UW-Madison Goals for the last lecture you should

Decision Tree Learning: Part 2 CS 760@UW-Madison Goals for the last lecture you should

Decision Tree Learning: Part 2 CS 760@UW-Madison Goals for the last lecture you should understand the following concepts the decision tree representation the standard top-down approach to learning a tree Occams razor

505 views • 38 slides
Decision tree learning Aim: find a small tree consistent with the training examples Idea:

Decision tree learning Aim: find a small tree consistent with the training examples Idea:

Decision tree learning Aim: find a small tree consistent with the training examples Idea: (recursively) choose most significant attribute as root of (sub)tree function DTL ( examples, attributes, default ) returns a decision tree if examples

261 views • 13 slides
Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington

Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington

Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington State University Outline Decision tree representation ID3 learning algorithm Entropy and information gain Overfitting

540 views • 30 slides
Decision Tree R Greiner Cmput 466 / 551 Learning Decision Trees Def'n: Decision Trees

Decision Tree R Greiner Cmput 466 / 551 Learning Decision Trees Def'n: Decision Trees

HTF: 9.2 B: 14.4 R N , C h a p t e r 1 8 1 8 . 3 Decision Tree R Greiner Cmput 466 / 551 Learning Decision Trees Def'n: Decision Trees Algorithm for Learning Decision Trees Entropy, Inductive Bias (Occam's

1.43k views • 87 slides
Decision tree learning Andrea Passerini passerini@disi.unitn.it Machine Learning Decision trees

Decision tree learning Andrea Passerini passerini@disi.unitn.it Machine Learning Decision trees

Decision tree learning Andrea Passerini passerini@disi.unitn.it Machine Learning Decision trees Learning the concept Go to lesson OUTLOOK Rain Overcast Sunny TRANSPORTATION LESSON NO Uncovered Covered Theoretical Practical NO YES

393 views • 12 slides
Decision Tree Based Learning of Program Invariants Deepak DSouza Department of Computer

Decision Tree Based Learning of Program Invariants Deepak DSouza Department of Computer

Floyd-Hoare Style Verification Decision Tree Learning ICE Learning Proofs with Multiple Invariants Decision Tree Based Learning of Program Invariants Deepak DSouza Department of Computer Science and Automation Indian Institute of Science,

648 views • 37 slides
Learning Decision Trees Representation is a decision tree. Bias is towards simple decision

Learning Decision Trees Representation is a decision tree. Bias is towards simple decision

Learning Decision Trees Representation is a decision tree. Bias is towards simple decision trees. Search through the space of decision trees, from simple decision trees to more complex ones. Decision trees A decision tree is

484 views • 10 slides
1 Optimization in decision graphs Unfolding to decision tree Only option until Shachter

1 Optimization in decision graphs Unfolding to decision tree Only option until Shachter

Decision graphs II Influence Diagrams Advanced Herd Management Anders Ringgaard Kristensen Slide 1 Outline Optimization methods Decision tree Strong junction tree Single Policy Updating Decision node ordering Advantages and

467 views • 10 slides
Chapter 18 Learning from Observations Decision tree examples Additional source used in preparing

Chapter 18 Learning from Observations Decision tree examples Additional source used in preparing

Chapter 18 Learning from Observations Decision tree examples Additional source used in preparing the slides: Jean-Claude Latombes CS121 slides: robotics.stanford.edu/~latombe/cs121 1 Decision Trees A decision tree allows a

608 views • 33 slides
1 Optimization in decision graphs The repeated milk test problem Unfolding to decision tree The

1 Optimization in decision graphs The repeated milk test problem Unfolding to decision tree The

Outline Decision trees Optimization methods Decision tree Decision graphs II Strong junction tree (Limited Memory) Influence Diagrams Single Policy Updating Markov property versus no forgetting Advanced Herd Management

512 views • 4 slides
A Brief History of Decision Tree Implementation MAX AUSTIN Overview Famous Decision Tree

A Brief History of Decision Tree Implementation MAX AUSTIN Overview Famous Decision Tree

A Brief History of Decision Tree Implementation MAX AUSTIN Overview Famous Decision Tree Algorithms Chi-squared Automatic Interaction Detector (CHAID) Classification and Regression Tree (CART) Iterative Dichotomiser 3 (ID3)

114 views • 9 slides
Learning from Data: Decision Trees Amos Storkey, School of Informatics University of Edinburgh

Learning from Data: Decision Trees Amos Storkey, School of Informatics University of Edinburgh

Learning from Data: Decision Trees Amos Storkey, School of Informatics University of Edinburgh Semester 1, 2004 LfD 2004 Decision Tree Learning - Overview Decision tree representation ID3 learning algorithm Entropy, Information gain

586 views • 23 slides
Decision Tree CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2019

Decision Tree CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2019

Decision Tree CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2019 Decision tree } One of the most intuitive classifiers that is easy to understand and construct } However, it also works very (very) well }

453 views • 41 slides
Decision Tree Learning: Part 1 CS 760@UW-Madison Zoo of machine learning models Note: only a

Decision Tree Learning: Part 1 CS 760@UW-Madison Zoo of machine learning models Note: only a

Decision Tree Learning: Part 1 CS 760@UW-Madison Zoo of machine learning models Note: only a subset of ML methods Figure from scikit-learn.org Even a subarea has its own collection Figure from asimovinstitute.org The lectures organized

534 views • 34 slides
Learning Objectives At the end of the class you should be able to: show an example of

Learning Objectives At the end of the class you should be able to: show an example of

Learning Objectives At the end of the class you should be able to: show an example of decision-tree learning explain how to avoid overfitting in decision-tree learning explain the relationship between linear and logistic regression explain how

714 views • 40 slides
Introduction Decision Tree for PlayTennis (Mitchell) CSCE CSCE 478/878 478/878 Outlook

Introduction Decision Tree for PlayTennis (Mitchell) CSCE CSCE 478/878 478/878 Outlook

Introduction Decision Tree for PlayTennis (Mitchell) CSCE CSCE 478/878 478/878 Outlook Lecture 3: Lecture 3: Learning Learning Decision trees form a simple, easily-interpretable, Decision Decision Trees Trees hypothesis Stephen Scott

400 views • 4 slides
Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3

Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3

0. Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. PLAN DT Learning: Basic Issues 1. Concept learning:

468 views • 36 slides
Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3

Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3

0. Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. PLAN DT Learning: Basic Issues 1. Concept learning:

641 views • 52 slides
Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the overlearning problem: pruning ensemble learning boosting Agenda Nearest neighbor models Finding nearest neighbors with kd trees

493 views • 27 slides
Decision Tree Learning: Part 1 Yingyu Liang Computer Sciences 760 Fall 2017

Decision Tree Learning: Part 1 Yingyu Liang Computer Sciences 760 Fall 2017

Decision Tree Learning: Part 1 Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven, David Page, Jude

583 views • 30 slides
Decision Tree Learning-Inspired Dynamic Variable Ordering for the Weighted CSP Hong Xu Kexuan

Decision Tree Learning-Inspired Dynamic Variable Ordering for the Weighted CSP Hong Xu Kexuan

Decision Tree Learning-Inspired Dynamic Variable Ordering for the Weighted CSP Hong Xu Kexuan Sun Sven Koenig T. K. Satish Kumar hongx@usc.edu, kexuansu@usc.edu, skoenig@usc.edu, tkskwork@gmail.com May, 2020 University of Southern California

452 views • 26 slides
Lecture 24: Other (Non-linear) Classifjers: Decision Tree Learning, Boosting, and Support Vector

Lecture 24: Other (Non-linear) Classifjers: Decision Tree Learning, Boosting, and Support Vector

. . . . . . . . . . . . . . . . . Lecture 24: Other (Non-linear) Classifjers: Decision Tree Learning, Boosting, and Support Vector Classifjcation Instructor: Prof. Ganesh Ramakrishnan October 20, 2016 . . . . . . . . .

357 views • 35 slides

More recommend