decision trees protocols and the fourier entropy
play

Decision trees, protocols, and the Fourier Entropy-Influence - PowerPoint PPT Presentation

May 06, 2023 •583 likes •1.77k views

Decision trees, protocols, and the Fourier Entropy-Influence Conjecture Andrew Wan (Simons Institute) John Wright (CMU) Chenggang Wu (Tsinghua) Fourier basics Given a Boolean function , Fourier

  1. Our results If is computable by a read-k DT , then . If is computable DT with expected depth d , and satisfies , then . also proven by [CKLS13] The FEI + conjecture of [OT13] composes. also proven by [OT13]

  2. Our technique • Want to show for certain Boolean functions .

  3. Our technique • Want to show for certain Boolean functions . • Previous papers have studied the expression

  4. Our technique • Want to show for certain Boolean functions . • Previous papers have studied the expression • We instead take an information theoretic approach via the Shannon Source Coding Theorem .

  5. Shannon Source Coding Theorem Given a random variable , avg # of bits needed to communicate .

  6. Shannon Source Coding Theorem Given a random variable , avg # of bits needed to communicate . Thus, to show , we need to construct an efficient protocol for communicating the value of .

  7. Shannon Source Coding Theorem Given a random variable , avg # of bits needed to communicate . Thus, to show , we need to construct an efficient protocol for communicating the value of . ( efficient = bits on average)

  8. a protocol for read-k DTs

  9. Protocol for read-k DTs is computed by 0 0 1 1 0 1 0 1 0 1

  10. Protocol for read-k DTs is computed by (a read-2 DT) 0 0 1 1 0 1 0 1 0 1

  11. Protocol for read-k DTs is computed by 0 0 1 1 0 1 0 1 0 1

  12. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1

  13. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  14. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  15. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  16. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  17. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  18. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  19. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  20. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  21. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  22. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  23. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  24. Protocol for read-k DTs is computed by Key fact : if is in the support 0 of , then the coordinates of appear in a root-to-leaf path in 0 1 1 . 0 1 0 1 0 1 Some sets in the support of :

  25. Protocol for read-k DTs Given , what should our protocol output? 0 0 1 1 0 1 0 1 0 1

  26. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 0 1 1 0 1 0 1 0 1

  27. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 0 1 0 1 0 1

  28. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 0 1 0 1 0 1

  29. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 2.) output the path’s description : 0 1 0 1 0 1

  30. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 2.) output the path’s description : 0 1 0 1 0 1

  31. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 2.) output the path’s description : 3.) indicate which nodes fall in : 0 1 0 1 0 1

  32. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 2.) output the path’s description : 3.) indicate which nodes fall in : 0 1 0 1 0 1

  33. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 2.) output the path’s description : 3.) indicate which nodes fall in : 0 1 0 1 0 1

  34. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 2.) output the path’s description : 3.) indicate which nodes fall in : 0 1 0 1 0 1 4.) final output:

  35. Protocol for read-k DTs Given , what should our protocol output? 0 0 1 1 0 1 0 1 0 1

  36. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 0 1 1 0 1 0 1 0 1

  37. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 0 1 0 1 0 1

  38. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 lots of choices! 0 1 0 1 0 1

  39. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 lots of choices! 0 1 0 1 0 1

  40. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 lots of choices! 0 1 0 1 0 1

  41. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find a path containing 0 1 1 lots of choices! 0 1 0 1 0 1

  42. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find the shortest path 0 1 1 containing 0 1 0 1 0 1

  43. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find the shortest path 0 1 1 containing 0 1 0 1 0 1

  44. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find the shortest path 0 1 1 containing 2.) output the path’s description : 0 1 0 1 0 1

  45. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find the shortest path 0 1 1 containing 2.) output the path’s description : 0 1 0 1 0 1 3.) indicate which nodes fall in :

  46. Protocol for read-k DTs Given , what should our protocol output? e.g., 0 1.) find the shortest path 0 1 1 containing 2.) output the path’s description : 0 1 0 1 0 1 3.) indicate which nodes fall in : 4.) final output:

  47. Analysis of protocol • A decision tree should be arranged with the most influential variables near the top.

  48. Analysis of protocol • A decision tree should be arranged with the most influential variables near the top. • Since every path output is root-to-leaf, the variables near the top will contribute a lot of bits to the expectation.

  49. Analysis of protocol • A decision tree should be arranged with the most influential variables near the top. • Since every path output is root-to-leaf, the variables near the top will contribute a lot of bits to the expectation. • In summary, contributes a lot to the expectation ⇒ is near the top of the tree ⇒ is highly influential

  50. A bad tree Let be a decision tree.

  51. A bad tree Let be a decision tree.

Recommend

Decision Tree R Greiner Cmput 466 / 551 Learning Decision Trees Def'n: Decision Trees

Decision Tree R Greiner Cmput 466 / 551 Learning Decision Trees Def'n: Decision Trees

HTF: 9.2 B: 14.4 R N , C h a p t e r 1 8 1 8 . 3 Decision Tree R Greiner Cmput 466 / 551 Learning Decision Trees Def'n: Decision Trees Algorithm for Learning Decision Trees Entropy, Inductive Bias (Occam's

1.43k views • 87 slides
IAML: Decision Trees Chris Williams and Victor Lavrenko School of Informatics Semester 1 1 / 17

IAML: Decision Trees Chris Williams and Victor Lavrenko School of Informatics Semester 1 1 / 17

IAML: Decision Trees Chris Williams and Victor Lavrenko School of Informatics Semester 1 1 / 17 Outline Decision trees: the idea Examples Top-down induction of decision trees Entropy Information gain Overfitting and

205 views • 17 slides
Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabs Pczos Contents

Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabs Pczos Contents

Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabs Pczos Contents Decision Trees: Definition + Motivation Algorithm for Learning Decision Trees Entropy, Mutual Information, Information gain Generalizations

1.06k views • 68 slides
Decision Trees Lecture 22 To left or to right 1 Decision Trees 2 Decision Trees A different

Decision Trees Lecture 22 To left or to right 1 Decision Trees 2 Decision Trees A different

Decision Trees Lecture 22 To left or to right 1 Decision Trees 2 Decision Trees A different complexity measure 2 Decision Trees A different complexity measure Number of bits of input read 2 Decision Trees A different complexity measure

1.62k views • 107 slides
Decision Trees Lecture 23 To left or to right 1 Decision Trees 2 Decision Trees A different

Decision Trees Lecture 23 To left or to right 1 Decision Trees 2 Decision Trees A different

Decision Trees Lecture 23 To left or to right 1 Decision Trees 2 Decision Trees A different complexity measure 2 Decision Trees A different complexity measure Number of bits of input read 2 Decision Trees A different complexity measure

1.3k views • 107 slides
Lecture 23: Decision Trees Decision trees Prof. Julia Hockenmaier

Lecture 23: Decision Trees Decision trees Prof. Julia Hockenmaier

CS440/ECE448: Intro to Artificial Intelligence Lecture 23: Decision Trees Decision trees Prof. Julia Hockenmaier juliahmr@illinois.edu http://cs.illinois.edu/fa11/cs440 Decision trees Decision tree

338 views • 7 slides
1 Baysian networks for decision analysis The TV show problem as a BN Problems More than

1 Baysian networks for decision analysis The TV show problem as a BN Problems More than

Several names Decision graphs Decision trees Influence diagrams Decision graphs I A certain kind of strictly symmetric decision trees Decisions and utilities No forgetting assumption LIMIDs Limited Memory

596 views • 6 slides
1 Baysian networks for decision analysis The TV show problem as a BN Problems Choice 1

1 Baysian networks for decision analysis The TV show problem as a BN Problems Choice 1

Several names Decision graphs Decision trees Influence diagrams Decision graphs I A certain kind of strictly symmetric decision trees Decisions and utilities No forgetting assumption LIMIDs Limited Memory

185 views • 6 slides
Learning Decision Trees Representation is a decision tree. Bias is towards simple decision

Learning Decision Trees Representation is a decision tree. Bias is towards simple decision

Learning Decision Trees Representation is a decision tree. Bias is towards simple decision trees. Search through the space of decision trees, from simple decision trees to more complex ones. Decision trees A decision tree is

484 views • 10 slides
Decision Tree Decision Trees A decision tree is a decision support tool that uses a tree-like

Decision Tree Decision Trees A decision tree is a decision support tool that uses a tree-like

Decision Tree Decision Trees A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible MSE 2400 EaLiCaRA consequences, including chance event Dr. Tom Way outcomes, resource costs, and

568 views • 8 slides
Learning from Data: Decision Trees Amos Storkey, School of Informatics University of Edinburgh

Learning from Data: Decision Trees Amos Storkey, School of Informatics University of Edinburgh

Learning from Data: Decision Trees Amos Storkey, School of Informatics University of Edinburgh Semester 1, 2004 LfD 2004 Decision Tree Learning - Overview Decision tree representation ID3 learning algorithm Entropy, Information gain

586 views • 23 slides
Tree Computation for Ranking and Classification CS240A, T. Yang, 2016 Outlines Decision Trees

Tree Computation for Ranking and Classification CS240A, T. Yang, 2016 Outlines Decision Trees

Tree Computation for Ranking and Classification CS240A, T. Yang, 2016 Outlines Decision Trees Learning Assembles: Random forest, boosted trees Decision Trees Decision trees can express any function of the input attributes.

452 views • 31 slides
Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of a discrete random variable. Properties: H(x) >= 0 Entropy Entropy Lesser the probability for an event, larger the entropy.

471 views • 21 slides
15-388/688 - Practical Data Science: Decision trees and interpretable models J. Zico Kolter

15-388/688 - Practical Data Science: Decision trees and interpretable models J. Zico Kolter

15-388/688 - Practical Data Science: Decision trees and interpretable models J. Zico Kolter Carnegie Mellon University Spring 2018 1 Outline Decision trees Training (classification) decision trees Interpreting predictions Boosting

381 views • 37 slides
Machine Learning III: Beyond Decision Trees Extensions to Decision Trees AI Class 15 (Ch.

Machine Learning III: Beyond Decision Trees Extensions to Decision Trees AI Class 15 (Ch.

Todays Class Machine Learning III: Beyond Decision Trees Extensions to Decision Trees AI Class 15 (Ch. 20.120.2) Sources of error Evaluating learned models B E Bayesian Learning E [1] B [1] A [1] C [1]

137 views • 3 slides
Medical Decision Making Learning: Decision Trees Artificial Intelligence CSPP 56553 February

Medical Decision Making Learning: Decision Trees Artificial Intelligence CSPP 56553 February

Medical Decision Making Learning: Decision Trees Artificial Intelligence CSPP 56553 February 11, 2004 Agenda Decision Trees: Motivation: Medical Experts: Mycin Basic characteristics Sunburn example From trees to rules

578 views • 28 slides
Decision Trees 2-26-16 Reading Quiz Decision trees are an algorithm for which machine learning

Decision Trees 2-26-16 Reading Quiz Decision trees are an algorithm for which machine learning

Decision Trees 2-26-16 Reading Quiz Decision trees are an algorithm for which machine learning task? a) clustering b) dimensionality reduction c) classification d) regression Reading Quiz Which error metric is most appropriate for

483 views • 14 slides
Decision Trees + k-Nearest Neighbors Matt Gormley Lecture 3 January 24, 2018 1 Q&A Q:

Decision Trees + k-Nearest Neighbors Matt Gormley Lecture 3 January 24, 2018 1 Q&A Q:

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Decision Trees + k-Nearest Neighbors Matt Gormley Lecture 3 January 24, 2018 1 Q&A Q: Why dont my entropy

582 views • 29 slides
Machine Learning and Data Mining Decision Trees Kalev Kask Decision trees Functional form

Machine Learning and Data Mining Decision Trees Kalev Kask Decision trees Functional form

+ Machine Learning and Data Mining Decision Trees Kalev Kask Decision trees Functional form f(x; ): nested if -then- else statements Discrete features: fully expressive (any function) Structure: Internal nodes: check

303 views • 27 slides
Wrap Up! Lecture 25 Decision Trees & Branching Programs Many Topics Not Covered! Decision

Wrap Up! Lecture 25 Decision Trees & Branching Programs Many Topics Not Covered! Decision

Wrap Up! Lecture 25 Decision Trees & Branching Programs Many Topics Not Covered! Decision Trees Another model of non-uniform computation A full binary tree with each internal Q 0 node labelled with an elementary boolean function of

490 views • 19 slides
CSC 411 Lecture 3: Decision Trees Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 3: Decision Trees Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 3: Decision Trees Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 03-Decision Trees 1 / 33 Today Decision Trees Simple but powerful learning algorithm One of the most

1.09k views • 33 slides
Optimal Sparse Decision Trees Xiyang Hu Cynthia Rudin Margo Seltzer Carnegie Mellon Duke

Optimal Sparse Decision Trees Xiyang Hu Cynthia Rudin Margo Seltzer Carnegie Mellon Duke

Optimal Sparse Decision Trees Xiyang Hu Cynthia Rudin Margo Seltzer Carnegie Mellon Duke University University of British University Columbia Decision Trees Decision Trees Decision Trees Should I click on the link in this email? Do I

589 views • 35 slides
Decision Trees Thomas Schwarz, SJ Decision Trees One of many machine learning methods

Decision Trees Thomas Schwarz, SJ Decision Trees One of many machine learning methods

Decision Trees Thomas Schwarz, SJ Decision Trees One of many machine learning methods Used to learn categories Example: The Iris Data Set Four measurements of flowers Learn how to predict species from measurement Iris Data

405 views • 40 slides
Decision trees PRISM - Nicolas Sutton-Charani 20/01/2020 N. Sutton-Charani Artificial

Decision trees PRISM - Nicolas Sutton-Charani 20/01/2020 N. Sutton-Charani Artificial

Decision trees PRISM - Nicolas Sutton-Charani 20/01/2020 N. Sutton-Charani Artificial intelligence Decision trees 1 / 47 1. Introduction 2. Use of decision trees 2.1 Prediction 2.2 Interpretability : Descriptive data analysis 3. Learning

478 views • 47 slides

More recommend