a fresh look at the bayes theorem from information theory
play

A Fresh Look at the Bayes Theorem from Information Theory Tan - PowerPoint PPT Presentation

Sep 30, 2022 •362 likes •1.13k views

A Fresh Look at the Bayes Theorem from Information Theory Tan Bui-Thanh Computational Engineering and Optimization (CEO) Group Department of Aerospace Engineering and Engineering Mechanics Institute for Computational Engineering and Sciences

  1. A Fresh Look at the Bayes’ Theorem from Information Theory Tan Bui-Thanh Computational Engineering and Optimization (CEO) Group Department of Aerospace Engineering and Engineering Mechanics Institute for Computational Engineering and Sciences (ICES) The University of Texas at Austin Babuska Series, ICES Sep 9, 2016 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 1 / 28

  2. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 2 / 28

  3. Large-scale computation under uncertainty Inverse electromagnetic scattering Randomness Random errors in measurements are unavoidable Inadequacy of the mathematical model (Maxwell equations) Challenge How to invert for the invisible shape/medium using computational � 10 6 � electromagnetics with O degree of freedoms? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 3 / 28

  4. Large-scale computation under uncertainty Full wave form seismic inversion Randomness Random errors in seismometer measurements are unavoidable Inadequacy of the mathematical model (elastodynamics) Challenge How to image the earth interior using forward computational model with � 10 9 � with O degree of freedoms? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 4 / 28

  5. Inverse Shape Electromagnetic Scattering Problem Maxwell Equations: r ⇥ E = � µ @ H @ t , (Faraday) r ⇥ H = ✏@ E @ t , (Ampere) E : Electric field, H : Magnetic field, µ : permeability, ✏ : permittivity (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

  6. Inverse Shape Electromagnetic Scattering Problem Maxwell Equations: r ⇥ E = � µ @ H @ t , (Faraday) r ⇥ H = ✏@ E @ t , (Ampere) E : Electric field, H : Magnetic field, µ : permeability, ✏ : permittivity Forward problem (discontinuous Galerkin discretization) d = G ( x ) where G maps shape parameters x to electric/magnetic field d at the measurement points (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

  7. Inverse Shape Electromagnetic Scattering Problem Maxwell Equations: r ⇥ E = � µ @ H @ t , (Faraday) r ⇥ H = ✏@ E @ t , (Ampere) E : Electric field, H : Magnetic field, µ : permeability, ✏ : permittivity Forward problem (discontinuous Galerkin discretization) d = G ( x ) where G maps shape parameters x to electric/magnetic field d at the measurement points Inverse Problem Given (possibly noise-corrupted) measurements on d , infer x ? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 5 / 28

  8. The Bayesian Statistical Inversion Framework (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

  9. The Bayesian Statistical Inversion Framework (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

  10. The Bayesian Statistical Inversion Framework Bayes Theorem ⇡ post ( x | d ) / ⇡ like ( d | x ) ⇥ ⇡ prior ( x ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 6 / 28

  11. Bayes theorem for inverse electromagnetic scattering Prior knowledge: The obstacle is smooth: Z 2 π ✓ ◆ ⇡ pr ( x ) / exp � � r 00 ( x ) d ✓ 0 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 7 / 28

  12. Bayes theorem for inverse electromagnetic scattering Prior knowledge: The obstacle is smooth: Z 2 π ✓ ◆ ⇡ pr ( x ) / exp � � r 00 ( x ) d ✓ 0 Likelihood: Additive Gaussian noise, for example, ✓ ◆ � 1 2 k G ( x ) � d k 2 ⇡ like ( d | x ) / exp C noise (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 7 / 28

  13. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 8 / 28

  14. Entropy Definition We define the uncertainty in a random variable X distributed by 0  ⇡ ( x )  1 as Z H ( X ) = � ⇡ ( x ) log ⇡ ( x ) dx � 0 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 9 / 28

  15. Entropy (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  16. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  17. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu Wiener: “...for it belongs to the two of us equally” (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  18. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu Wiener: “...for it belongs to the two of us equally” Shannon: “...a mathematical pun” (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  19. Entropy Wiener and Shannon Kolmogorov Copied from Sergio Verdu Wiener: “...for it belongs to the two of us equally” Shannon: “...a mathematical pun” Kolmogorov: “...has no physical interpretation” (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 10 / 28

  20. Entropy Entropy of uniform distribution (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  21. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  22. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 ⇡ ( u ) := 1 | X | ) H ( U ) = log ( | X | ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  23. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 ⇡ ( u ) := 1 | X | ) H ( U ) = log ( | X | ) How uncertain is the uniform random variable? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  24. Entropy Entropy of uniform distribution Let U be a uniform random variable with values in X , and | X | < 1 ⇡ ( u ) := 1 | X | ) H ( U ) = log ( | X | ) How uncertain is the uniform random variable? H ( X )  H ( U ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 11 / 28

  25. 100 years of uniform distribution source: Christoph Aistleitner (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 12 / 28

  26. 100 years of uniform distribution source: Christoph Aistleitner Hermann Weyl (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 12 / 28

  27. and Maximum entropy Maximum entropy distribution X with known mean and variance (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  28. and Maximum entropy Maximum entropy distribution X with known mean and variance ⇡ ( x )? with maximum entropy (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  29. and Maximum entropy Maximum entropy distribution X with known mean and variance ⇡ ( x )? with maximum entropy Z max π ( x ) H ( X ) = � ⇡ ( x ) log( ⇡ ( x )) dx subject to Z x ⇡ ( x ) dx = µ Z ( x � µ ) 2 ⇡ ( x ) dx = � 2 Z ⇡ ( x ) dx = 1 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  30. Gaussian and Maximum entropy Maximum entropy distribution X with known mean and variance ⇡ ( x )? with maximum entropy Z max π ( x ) H ( X ) = � ⇡ ( x ) log( ⇡ ( x )) dx subject to Z x ⇡ ( x ) dx = µ Z ( x � µ ) 2 ⇡ ( x ) dx = � 2 Z ⇡ ( x ) dx = 1 � µ, � 2 � Gaussian distribution: ⇡ ( x ) = N (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 13 / 28

  31. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 14 / 28

  32. Relative Entropy Abraham Wald (1945) Harold Je ff reys (1945) Z ✓ ⇡ ( x ) ◆ D ( ⇡ || q ) := ⇡ ( x ) log dx q ( x ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 15 / 28

  33. Kullback-Leibler divergence = Relative Entropy Solomon Kullback Richard Leibler (1951) (1951) ✓ ⇡ ( x ) ◆ Z D ( ⇡ || q ) := ⇡ ( x ) log dx q ( x ) (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 16 / 28

  34. Kullback-Leibler divergence = Relative Entropy Solomon Kullback Richard Leibler (1951) (1951) ✓ ⇡ ( x ) ◆ ✓ ⇡ i ◆ Z X dx discrete D ( ⇡ || q ) := ⇡ ( x ) log = ⇡ i log q ( x ) q i (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 16 / 28

  35. Information Inequality The most important inequality in information theory D ( ⇡ || q ) � 0 Can we see it easily? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 17 / 28

  36. Information Inequality The most important inequality in information theory D ( ⇡ || q ) � 0 Can we see it easily? (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 17 / 28

  37. Outline Bayesian Inversion Framework 1 Entropy 2 Relative Entropy 3 Bayes’ Theorem and Information Theory 4 Conclusions 5 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 18 / 28

  38. From Relative Entropy to Bayes’ Theorem Toss n times an k th dimensional dice with the prior distribution of k X each face { p i } k i =1 : p i = 1 i =1 (CEO Group UT Austin) Bayes’ Theorem Bayesian Inversions 19 / 28

Recommend

Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for

Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for

Nave Bayes Classifica/on Debapriyo Majumdar Data Mining Fall 2014 Indian Statistical Institute Kolkata August 14, 2014 Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for two

1.04k views • 20 slides
Statistical Reasoning Probability and Bayes' Theorem Certainty Factors and Rule- Based

Statistical Reasoning Probability and Bayes' Theorem Certainty Factors and Rule- Based

Statistical Reasoning Probability and Bayes' Theorem Certainty Factors and Rule- Based Systems Bayesian Networks Dempster-Shafer Theory Fuzzy Logic Chapter 8 1 Probability and Bayes' Theorem P(H i|E) = the probability that

281 views • 13 slides
Chapter 3: Basics from Probability Theory and Statistics 3.1 Probability Theory Events,

Chapter 3: Basics from Probability Theory and Statistics 3.1 Probability Theory Events,

Chapter 3: Basics from Probability Theory and Statistics 3.1 Probability Theory Events, Probabilities, Bayes Theorem, Random Variables, Distributions, Moments, Tail Bounds, Central Limit Theorem, Entropy Measures 3.2 Statistical Inference

845 views • 45 slides
Formal Modeling in Cognitive Science Independence Lecture 23: Conditional Probability; Bayes

Formal Modeling in Cognitive Science Independence Lecture 23: Conditional Probability; Bayes

Conditional Probability and Independence Conditional Probability and Independence Bayes Theorem Bayes Theorem An Application of Bayes Theorem An Application of Bayes Theorem 1 Conditional Probability and Independence Conditional

247 views • 6 slides
Frequentist example An entomologist spots what might be a rare subspecies of beetle, due to the

Frequentist example An entomologist spots what might be a rare subspecies of beetle, due to the

CS201 Bayes Theorem Excerpts from WikiPedia http://en.wikipedia.org/wiki/Bayes%27_theorem http://en.wikipedia.org/wiki/Bayesian_inference Bayes's theorem is stated mathematically as the following simple form: [1] For an epistemological

333 views • 5 slides
1 Problem with Brute force Nave Bayes ( ) ( ) ( ) It cannot generalize to unseen

1 Problem with Brute force Nave Bayes ( ) ( ) ( ) It cannot generalize to unseen

Outline Bayes theorem Discrete Bayesian classifiers Maximum likelihood classification Brute force Bayesian learning Nave Bayes Lecture 5 Bayesian Belief Networks Bayes theorem Maximum likelihood ( ) ( ) ( ) P x

72 views • 4 slides
Spam Filtering with Naive Bayes Classifier Yuriy Arabskyy June 6, 2017 Table of contents What

Spam Filtering with Naive Bayes Classifier Yuriy Arabskyy June 6, 2017 Table of contents What

Spam Filtering with Naive Bayes Classifier Yuriy Arabskyy June 6, 2017 Table of contents What is spam? Different spam types Anti-Spam Techniques Probability theory basics Conditional probability Bayes Theorem Naive Bayes Theorem Spam

1.04k views • 27 slides
Bayesian Learning Bayes Theorem MAP, ML hypotheses MAP learners Minimum description

Bayesian Learning Bayes Theorem MAP, ML hypotheses MAP learners Minimum description

Bayesian Learning Bayes Theorem MAP, ML hypotheses MAP learners Minimum description length principle Bayes optimal classifier Naive Bayes learner Example: Learning over text data Bayesian belief networks

1.11k views • 51 slides
Which probability Which probability Which probability Which probability theory for cosmology?

Which probability Which probability Which probability Which probability theory for cosmology?

Which probability Which probability Which probability Which probability theory for cosmology? theory for cosmology? theory for cosmology? theory for cosmology? From Bayes theorem to the anthropic principle From Bayes theorem to the

537 views • 35 slides
Conditional and Small Sample Probability August 6, 2019 August 6, 2019 1 / 63 Bayes Theorem

Conditional and Small Sample Probability August 6, 2019 August 6, 2019 1 / 63 Bayes Theorem

Conditional and Small Sample Probability August 6, 2019 August 6, 2019 1 / 63 Bayes Theorem Bayes Theorem will help us more easily calculate P (statement about variable 1 | statement about variable 2) when we have information about P

759 views • 63 slides
Formal Modeling in Cognitive Science Lecture 18: Conditional Probability; Bayes Theorem Steve

Formal Modeling in Cognitive Science Lecture 18: Conditional Probability; Bayes Theorem Steve

Conditional Probability and Independence Bayes Theorem Formal Modeling in Cognitive Science Lecture 18: Conditional Probability; Bayes Theorem Steve Renals (notes by Frank Keller) School of Informatics University of Edinburgh

169 views • 13 slides
Data Mining Lecture 06: Bayes Theorem Theses slides are based on the slides by Tan,

Data Mining Lecture 06: Bayes Theorem Theses slides are based on the slides by Tan,

CISC 4631 Data Mining Lecture 06: Bayes Theorem Theses slides are based on the slides by Tan, Steinbach and Kumar (textbook authors) Eamonn Koegh (UC Riverside) Andrew Moore (CMU/Google) 1 Nave Bayes Classifier Thomas

657 views • 36 slides
Bayes Theorem Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] Note: This

Bayes Theorem Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] Note: This

Bayes Theorem Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] Note: This is very useful in both this course and in life. Example of Application of Bayess theorem Pr [ A | B ] = Pr [ B | A ] Pr [ A ] Pr [ B ] . There

86 views • 8 slides
1 Bayes Theorem Bayesian Categorization Determine category of x k by determining for each y i

1 Bayes Theorem Bayesian Categorization Determine category of x k by determining for each y i

Axioms of Probability Theory All probabilities between 0 and 1 P A 0 ( ) 1 CS 391L: Machine Learning: True proposition has probability 1, false has Bayesian Learning: probability 0. Nave Bayes P(true) = 1

765 views • 4 slides
PCP Theorem [PCP Theorem is] the most important result in complexity theory since Cooks

PCP Theorem [PCP Theorem is] the most important result in complexity theory since Cooks

PCP Theorem [PCP Theorem is] the most important result in complexity theory since Cooks Theorem. Ingo Wegener, 2005 Computational Complexity, by Fu Yuxi PCP Theorem 1 / 119 S. Arora, C. Lund, R. Motwani, M. Sudan, and M. Szegedy. Proof

1.53k views • 120 slides
Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine

Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine

Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck http://cseweb.ucsd.edu/classes/sp16/cse21-bd/ June 1, 2016 Independence Rosen p. 486 Theorem : If X and Y are independent random

1.26k views • 92 slides
Statistical classification Lecture notes Naive Bayes Bayes' theorem P ( c|a ) P ( a ) = P ( a|c

Statistical classification Lecture notes Naive Bayes Bayes' theorem P ( c|a ) P ( a ) = P ( a|c

Statistical classification Lecture notes Naive Bayes Bayes' theorem P ( c|a ) P ( a ) = P ( a|c ) P (c) P ( c|f ) % of class c given feature(s) a Posterior This will be our target P ( f|c ) % of feature f given class c

676 views • 36 slides
Unit2: Probabilityanddistributions 2. Bayes theorem and Bayesian inference If you

Unit2: Probabilityanddistributions 2. Bayes theorem and Bayesian inference If you

Announcements Unit2: Probabilityanddistributions 2. Bayes theorem and Bayesian inference If you received an email from me about your clicker registration being missing and you still have not given me your info on the Sta 101 - Fall

298 views • 4 slides
Another Walkthrough of Variational Bayes Bevan Jones ML for NLP Reading Group The University of

Another Walkthrough of Variational Bayes Bevan Jones ML for NLP Reading Group The University of

Another Walkthrough of Variational Bayes Bevan Jones ML for NLP Reading Group The University of Edinburgh 18 th of October, 2012 2 Variational Bayes? Bayes Bayes Theorem But the integral is intractable! Sampling Gibbs,

1.51k views • 140 slides
Unit2: Probabilityanddistributions 2. Bayes theorem and Bayesian inference Sta 101 - Spring

Unit2: Probabilityanddistributions 2. Bayes theorem and Bayesian inference Sta 101 - Spring

Announcements Unit2: Probabilityanddistributions 2. Bayes theorem and Bayesian inference Sta 101 - Spring 2017 PS 2 is posted Start reviewing Unit 3 materials Duke University, Department of Statistical Science Dr. Mukherjee Slides

354 views • 4 slides
Formal Modeling in Cognitive Science Lecture 19: Application of Bayes Theorem; Discrete Random

Formal Modeling in Cognitive Science Lecture 19: Application of Bayes Theorem; Discrete Random

Application of Bayes Theorem Discrete Random Variables Distributions Formal Modeling in Cognitive Science Lecture 19: Application of Bayes Theorem; Discrete Random Variables; Distributions Steve Renals (notes by Frank Keller) School of

453 views • 21 slides
La th eorie PAC-Bayes en apprentissage supervis e Pr esentation au LRI de luniversit

La th eorie PAC-Bayes en apprentissage supervis e Pr esentation au LRI de luniversit

The mathematics of the PAC-Bayes Theory PAC-Bayes bounds and algorithms References La th eorie PAC-Bayes en apprentissage supervis e Pr esentation au LRI de luniversit e Paris XI Fran cois Laviolette, Laboratoire du GRAAL,

615 views • 50 slides
Math 186: Conditional Probability and Bayes Theorem (2.4) Independence (2.5) Math 283: Ewens

Math 186: Conditional Probability and Bayes Theorem (2.4) Independence (2.5) Math 283: Ewens

Math 186: Conditional Probability and Bayes Theorem (2.4) Independence (2.5) Math 283: Ewens &amp; Grant 1.12.45 Prof. Tesler Math 186 and 283 Fall 2019 Prof. Tesler Conditional Probability and Bayes Theorem Math 186 &amp; 283 /

499 views • 38 slides
Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine

Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine

Independence, Variance, Bayes Theorem Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck http://cseweb.ucsd.edu/classes/sp16/cse21-bd/ May 16, 2016 Resolving collisions with chaining Hash Table Each memory location holds

770 views • 60 slides

More recommend