na ve bayesian learning
play

Nave Bayesian Learning Sven Koenig, USC Russell and Norvig, 3 rd - PDF document

Nov 10, 2023 •364 likes •427 views

12/18/2019 Nave Bayesian Learning Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections 13.5.2 and 20.2.2 These slides are new and can contain mistakes and typos. Please report them to Sven (skoenig@usc.edu). 1 Nave Bayesian Learning

  1. 12/18/2019 Naïve Bayesian Learning Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections 13.5.2 and 20.2.2 These slides are new and can contain mistakes and typos. Please report them to Sven (skoenig@usc.edu). 1 Naïve Bayesian Learning • We now apply what we have learned to machine learning. 2 1

  2. 12/18/2019 Inductive Learning for Classification • Labeled examples Feature_1 Feature_2 Class true true true true false false false true false Learn f(Feature_1, Feature_2) = Class from f(true, true) = true f(true, false) = false f(false, true) = false The function needs to be consistent with all labeled examples and should make the fewest mistakes on the unlabeled examples. • Unlabeled examples Feature_1 Feature_2 Class false false ? true true ? 3 Naïve Bayesian Learning • Assume that the features are conditionally independent of each other given the class. • This naïve (= potentially wrong) assumption keeps the number of parameters to be learned small. Class … Feature_1 Feature_n Naïve Bayesian Network 4 2

  3. 12/18/2019 Naïve Bayesian Learning • Use maximum-likelihood estimates to learn the probabilities in the conditional probability tables from the labeled examples, that is, use frequencies to estimate the probabilities. Feature_1 Feature_2 Class true true true true false false false true false P(Class) Class 1/3 There are two examples whose class is false: Feature_1 of one of it is true and Feature_1 of the other one of it is false. Class P(Feature_1| Class) Class P(Feature_2 | Class) true 1 Feature_1 Feature_2 true 1 false 1/2 false 1/2 5 Naïve Bayesian Learning • Calculate the probabilities of the class values given the feature values for unlabeled examples Feature_1 Feature_2 Class false false ? P(Class) Class 1/3 Class P(Feature_1| Class) Class P(Feature_2 | Class) true 1 Feature_1 Feature_2 true 1 false 1/2 false 1/2 • Either make a probabilistic prediction by outputting P(Class | NOT Feature_1, NOT Feature_2) or a deterministic prediction by outputting the more likely class. 6 3

  4. 12/18/2019 Naïve Bayesian Learning • P(Class, NOT Feature_1, NOT Feature_2) = P(Class) P(NOT Feature_1 | Class) P(NOT Feature_2 | Class) = 1/3 0 0 = 0 • P(NOT Class, NOT Feature_1, NOT Feature_2) = P(NOT Class) P(NOT Feature_1 | NOT Class) P(NOT Feature_2 | NOT Class) = 2/3 1/2 1/2 = 1/6 • P(NOT Feature_1, NOT Feature_2) = P(Class, NOT Feature_1, NOT Feature_2) + P(NOT Class, NOT Feature_1, NOT Feature_2) = 0 + 1/6 = 1/6 • P(Class | NOT Feature_1, NOT Feature_2) = P(Class, NOT Feature_1, NOT Feature_2) / P(NOT Feature_1, NOT Feature_2) = 0 / (1/6) = 0 • P(NOT Class | NOT Feature_1, NOT Feature_2) = P(NOT Class, NOT Feature_1, NOT Feature_2) / P(NOT Feature_1, NOT Feature_2) = (1/6) / (1/6) = 1 Feature_1 Feature_2 Class false false P(Class | NOT Feature_1, NOT Feature_2) = 0 or false 7 Naïve Bayesian Learning • Calculate the probabilities of the class values given the feature values for unlabeled examples Feature_1 Feature_2 Class true true ? P(Class) Class 1/3 Class P(Feature_1| Class) Class P(Feature_2 | Class) true 1 Feature_1 Feature_2 true 1 false 1/2 false 1/2 • Either make a probabilistic prediction by outputting P(Class | Feature_1, Feature_2) or a deterministic prediction by outputting the more likely class. 8 4

  5. 12/18/2019 Naïve Bayesian Learning • P(Class, Feature_1, Feature_2) = P(Class) P(Feature_1 | Class) P(Feature_2 | Class) = 1/3 1 1 = 1/3 • P(NOT Class, Feature_1, Feature_2) = P(NOT Class) P(Feature_1 | NOT Class) P(Feature_2 | NOT Class) = 2/3 1/2 1/2 = 1/6 • P(Feature_1, Feature_2) = P(Class, Feature_1, Feature_2) + P(NOT Class, Feature_1, Feature_2) = 1/3 + 1/6 = 1/2 • P(Class | Feature_1, Feature_2) = P(Class, Feature_1, Feature_2) / P(Feature_1, Feature_2) = (1/3) / (1/2) = 2/3 • P(NOT Class | Feature_1, Feature_2) = P(NOT Class, Feature_1, Feature_2) / P(Feature_1, Feature_2) = (1/6) / (1/2) = 1/3 Feature_1 Feature_2 Class true true P(Class | Feature_1, Feature_2) = 2/3 or true 9 Naïve Bayesian Learning • For inductive learning, we typically demand that the learned function is consistent with all labeled examples (if possible). However, then we should have calculated P(Class | Feature_1, Feature_2) = 1. • This is not possible because the naïve Bayesian assumption does not hold for the labeled examples (see next slide). • Thus, a naïve Bayesian network cannot represent the labeled examples correctly and thus cannot represent all Boolean functions correctly. • Just like for single perceptrons, this does not mean that they should not be used. They will make some mistakes for some Boolean functions but they often work well, that is, make few mistakes on the labeled and unlabeled examples. 10 5

  6. 12/18/2019 Naïve Bayesian Learning • The assumption that the features are conditionally independent of each other given the class does not hold for the labeled examples. Feature_1 Feature_2 Class true true true true false false false true false • For example, P(Feature_1 | NOT Class) = 1/2 but P(Feature_1 | Feature_2, NOT Class) = 0. 11 Naïve Bayesian Learning • Properties (some versus decision trees) • Are very tolerant of noise in feature and class values of examples • Can make deterministic or probabilistic predictions • Learn quickly even for large problems • Cannot represent all Boolean functions (since the naïve Bayesian assumption does not hold for all of them) • Early application • Email spam detectors (where Feature_i = “How often does the i th word in a dictionary appear in the email?” and Class = “Is the email spam?”) 12 6

Recommend

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression Bayesian Gaussian Mixture Models Non-parametric Bayes 2 Take Away ... 1. Maximum Likelihood Estimate (MLE) = arg max p ( D| )

609 views • 27 slides
Outline Intro to RL and Bayesian Learning History of Bayesian RL Model-based Bayesian

Outline Intro to RL and Bayesian Learning History of Bayesian RL Model-based Bayesian

Outline Intro to RL and Bayesian Learning History of Bayesian RL Model-based Bayesian RL Prior knowledge, policy optimization, discussion, Bayesian approaches for other RL variants Model-free Bayesian RL Gaussian

1.23k views • 63 slides
CS440/ECE448 Lecture 15: Bayesian Inference and Bayesian Learning Slides by Svetlana Lazebnik,

CS440/ECE448 Lecture 15: Bayesian Inference and Bayesian Learning Slides by Svetlana Lazebnik,

CS440/ECE448 Lecture 15: Bayesian Inference and Bayesian Learning Slides by Svetlana Lazebnik, 10/2016 Modified by Mark Hasegawa-Johnson, 3/2019 Bayesian Inference and Bayesian Learning Bayes Rule Bayesian Inference Misdiagnosis

829 views • 38 slides
Learning Objectives At the end of the class you should be able to: derive Bayesian learning from

Learning Objectives At the end of the class you should be able to: derive Bayesian learning from

Learning Objectives At the end of the class you should be able to: derive Bayesian learning from first principles explain how the Beta and Dirichlet distributions are used for Bayesian learning. D. Poole and A. Mackworth 2019 c Artificial

272 views • 16 slides
Adversarial Approaches to Bayesian Learning and Bayesian Approaches to Adversarial Robustness

Adversarial Approaches to Bayesian Learning and Bayesian Approaches to Adversarial Robustness

Adversarial Approaches to Bayesian Learning and Bayesian Approaches to Adversarial Robustness Ian Goodfellow, OpenAI Research Scientist NIPS 2016 Workshop on Bayesian Deep Learning Barcelona, 2016-12-10 Speculation on Three Topics Can we

769 views • 21 slides
Beyond Uniform Priors in Bayesian Network Structure Learning (for Discrete Bayesian Networks)

Beyond Uniform Priors in Bayesian Network Structure Learning (for Discrete Bayesian Networks)

Beyond Uniform Priors in Bayesian Network Structure Learning (for Discrete Bayesian Networks) Marco Scutari scutari@stats.ox.ac.uk Department of Statistics University of Oxford April 5, 2017 Bayesian Network Structure Learning Learning a BN

536 views • 26 slides
Bayesian learning (with a recap of Bayesian Networks) Applied artificial intelligence (EDA132)

Bayesian learning (with a recap of Bayesian Networks) Applied artificial intelligence (EDA132)

Bayesian learning (with a recap of Bayesian Networks) Applied artificial intelligence (EDA132) Lecture 05 2016-02-02 Elin A. Topp Material based on course book, chapters 14.1-3, 20, and on Tom M. Mitchell, Machine Learning, McGraw-Hill,

557 views • 28 slides
Overview Bayesian Methods for Parameter Estimation Introduction to Bayesian Statistics: Learning

Overview Bayesian Methods for Parameter Estimation Introduction to Bayesian Statistics: Learning

Overview Bayesian Methods for Parameter Estimation Introduction to Bayesian Statistics: Learning a Probability Learning the mean of a Gaussian Chris Williams Readings: Bishop 2.1 (Beta), 2.2 (Dirichlet), 2.3.6 School of Informatics,

177 views • 5 slides
Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin

Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin

Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ BARK 2008 Some

456 views • 26 slides
Learning: A Bayesian solution Dmitry P. Vetrov Research professor at HSE, Head of Bayesian

Learning: A Bayesian solution Dmitry P. Vetrov Research professor at HSE, Head of Bayesian

Open Problems in Deep Learning: A Bayesian solution Dmitry P. Vetrov Research professor at HSE, Head of Bayesian methods research group http://bayesgroup.ru Idea of the talk 2 Deep Learning Revolution in machine learning Deep neural

452 views • 26 slides
CSCE 478/878 Lecture 6: Bayesian Learning MAP learners 1. Provide practical learning

CSCE 478/878 Lecture 6: Bayesian Learning MAP learners 1. Provide practical learning

Bayesian Methods Not all hypotheses are created equal (even if they are all Outline consistent with the training data) Might have reasons (domain information) to favor some Bayes Theorem hypotheses over others a priori Bayesian methods work

201 views • 6 slides
Probabilistic Graphical Models Lecture 5 Bayesian Learning of Bayesian Networks CS/CNS/EE

Probabilistic Graphical Models Lecture 5 Bayesian Learning of Bayesian Networks CS/CNS/EE

Probabilistic Graphical Models Lecture 5 Bayesian Learning of Bayesian Networks CS/CNS/EE 155 Andreas Krause Announcements Recitations: Every Tuesday 4-5:30 in 243 Annenberg Homework 1 out. Due in class Wed Oct 21 Project proposals due

521 views • 41 slides
2016-02-10 5.2 Learning Bayesian networks: Learning phase: General idea Representing and

2016-02-10 5.2 Learning Bayesian networks: Learning phase: General idea Representing and

2016-02-10 5.2 Learning Bayesian networks: Learning phase: General idea Representing and storing the knowledge See Witten et al. 2011. The network we want to learn has to have a node for every feature that we have, resp. want to be Bayesian

58 views • 4 slides
Function Space Priors in Bayesian Deep Learning Roger Grosse Motivation Today Bayesian deep

Function Space Priors in Bayesian Deep Learning Roger Grosse Motivation Today Bayesian deep

Function Space Priors in Bayesian Deep Learning Roger Grosse Motivation Today Bayesian deep learning is most often tested on regularization (Bayesian Occams Razor, description length regularization) smoothing the predictions

420 views • 25 slides
AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed Acyclic Graph (DAG)

AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed Acyclic Graph (DAG)

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed Acyclic Graph (DAG) Bayesian Networks General Factorization Bayesian Curve Fitting (1) Polynomial Bayesian Curve Fitting (2) Plate Bayesian

846 views • 71 slides
Foundations of AI Why learning works 1 6 . Statistical Machine Learning Bayesian Learning and

Foundations of AI Why learning works 1 6 . Statistical Machine Learning Bayesian Learning and

Contents Statistical learning Foundations of AI Why learning works 1 6 . Statistical Machine Learning Bayesian Learning and Why Learning Works W olfram Burgard, Bernhard Nebel, and Andreas Karw ath 10/ 1 10/ 2 Statistical Learning

305 views • 5 slides
Bayesian Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 6

Bayesian Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 6

0. Bayesian Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 6 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Two Roles for the Bayesian Methods in Learning 1. Provides

709 views • 54 slides
Bayesian Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 6

Bayesian Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 6

0. Bayesian Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 6 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Two Roles for the Bayesian Methods in Learning 1. Provides

1.15k views • 71 slides
Bayesian Reinforcement Learning: A Survey Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau,

Bayesian Reinforcement Learning: A Survey Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau,

Bayesian Reinforcement Learning: A Survey Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar Presented by Jacob Nogas ft. Animesh Garg (cameo) Bayesian RL: What - Leverage Bayesian Information in RL problem - Dynamics -

771 views • 52 slides
Graphical Models and Bayesian Networks Required reading: Ghahramani, section 2, Learning

Graphical Models and Bayesian Networks Required reading: Ghahramani, section 2, Learning

Graphical Models and Bayesian Networks Required reading: Ghahramani, section 2, Learning Dynamic Bayesian Networks (just 3.5 pages :-) Optional reading: Mitchell, chapter 6.11 Bayesian Belief Networks Machine Learning 10-701

548 views • 11 slides
5.2 Learning Bayesian networks: General idea See Witten et al. 2011. Bayesian (belief) networks

5.2 Learning Bayesian networks: General idea See Witten et al. 2011. Bayesian (belief) networks

5.2 Learning Bayesian networks: General idea See Witten et al. 2011. Bayesian (belief) networks aim at representing probability distributions of features and how to combine them to make predictions about the likelihood that a particular

622 views • 22 slides
Naive Bayesian Learning in Social Networks Jerry Anunrojwong (Harvard) joint with Nat Sothanaphan

Naive Bayesian Learning in Social Networks Jerry Anunrojwong (Harvard) joint with Nat Sothanaphan

Naive Bayesian Learning in Social Networks Jerry Anunrojwong (Harvard) joint with Nat Sothanaphan (MIT) EC18 Social Learning state of the world unknown to the agents Rule: can only talk to your neighbors Prior works: Bayesian Learning

730 views • 33 slides
Optimal Algorithms for Learning Bayesian Optimal Algorithms for Learning Bayesian Network

Optimal Algorithms for Learning Bayesian Optimal Algorithms for Learning Bayesian Network

Optimal Algorithms for Learning Bayesian Optimal Algorithms for Learning Bayesian Network Structures: Network Structures: Introduction and Heuristic Search Introduction and Heuristic Search Changhe Yuan UAI 2015 Tutorial Sunday, July 12 th ,

858 views • 56 slides
Overview Bayesian Model Selection Bayesian Learning of CPTs Dealing with Multiple Models Chris

Overview Bayesian Model Selection Bayesian Learning of CPTs Dealing with Multiple Models Chris

Overview Bayesian Model Selection Bayesian Learning of CPTs Dealing with Multiple Models Chris Williams Other Scores for Model Comparison Searching over Belief Network structures School of Informatics, University of Edinburgh Readings:

234 views • 6 slides

More recommend