Information Theory and Statistical Inference Samuel Cheng School of - PowerPoint PPT Presentation

Information Theory and Statistical Inference Samuel Cheng School of ECE University of Oklahoma August 23, 2018 S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 1 / 45

Lecture 2 Introduction to probabilistic inference Inference o : (Observed) evidence, θ : Parameter, x : prediction Maximum Likelihood (ML) x = arg max x p ( x | ˆ θ ) , ˆ ˆ θ = arg max θ p ( o | θ ) Maximum A Posteriori (MAP) x = arg max x p ( x | ˆ θ ) , ˆ ˆ θ = arg max θ p ( θ | o ) Bayesian � x = � ˆ x x p ( x | θ ) p ( θ | o ) θ � �� p ( x | o ) where p ( θ | o ) = p ( o | θ ) p ( θ ) ∝ p ( o | θ ) p ( θ ) p ( o ) �� prior S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 2 / 45

Lecture 2 Introduction to probabilistic inference Coin Flip C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 Which coin will I use? P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 Prior: Probability of a hypothesis before we make any observations (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 3 / 45

Lecture 2 Introduction to probabilistic inference Coin Flip C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 Which coin will I use? P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 Uniform Prior: All hypothesis are equally likely before we make any observations (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 4 / 45

Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = ? P(C 2 |H) = ? P(C 3 |H) = ? 3 P ( C 1 | H ) = P ( H | C 1 ) P ( C 1 ) � P ( H ) = P ( H | C i ) P ( C i ) P ( H ) i =1 C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 5 / 45

Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = 0.066 P(C 2 |H) = 0.333 P(C 3 |H) = 0.6 Posterior: Probability of a hypothesis given data C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 6 / 45

Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.21 P(C 2 |HT) = 0.58 P(C 3 |HT) = 0.21 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 9 / 45

Lecture 2 Introduction to probabilistic inference Your Estimate? What is the probability of heads after two experiments? Most likely coin: Best estimate for P(H) C 2 P(H|C 2 ) = 0.5 C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 1/3 P(C 2 ) = 1/3 P(C 3 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 10 / 45

Lecture 2 Introduction to probabilistic inference Your Estimate? Maximum Likelihood Estimate: The best hypothesis that fits observed data assuming uniform prior Most likely coin: Best estimate for P(H) C 2 P(H|C 2 ) = 0.5 C 2 P(H|C 2 ) = 0.5 P(C 2 ) = 1/3 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 11 / 45

Lecture 2 Introduction to probabilistic inference Using Prior Knowledge • Should we always use Uniform Prior? • Background knowledge: • Heads => you go first in Abalone against TA • TAs are nice people • => TA is more likely to use a coin biased in your favor C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 12 / 45

Lecture 2 Introduction to probabilistic inference Using Prior Knowledge We can encode it in the prior: P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 13 / 45

Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = ? P(C 2 |H) = ? P(C 3 |H) = ? P ( C 1 | H ) = α P ( H | C 1 ) P ( C 1 ) C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 14 / 45

Lecture 2 Introduction to probabilistic inference Experiment 1: Heads Which coin did I use? P(C 1 |H) = 0.006 P(C 2 |H) = 0.165 P(C 3 |H) = 0.829 ML posterior after Exp 1: P(C 1 |H) = 0.066 P(C 2 |H) = 0.333 P(C 3 |H) = 0.600 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 15 / 45

Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 P ( C 1 | HT ) = α P ( HT | C 1 ) P ( C 1 ) = α P ( H | C 1 ) P ( T | C 1 ) P ( C 1 ) C 2 C 3 C 1 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 17 / 45

Lecture 2 Introduction to probabilistic inference Experiment 2: Tails Which coin did I use? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 18 / 45

Lecture 2 Introduction to probabilistic inference Your Estimate? What is the probability of heads after two experiments? Most likely coin: Best estimate for P(H) C 3 P(H|C 3 ) = 0.9 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 2 ) = 0.5 P(H|C 3 ) = 0.9 P(C 1 ) = 0.05 P(C 2 ) = 0.25 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 19 / 45

Lecture 2 Introduction to probabilistic inference Your Estimate? Maximum A Posteriori (MAP) Estimate: The best hypothesis that fits observed data assuming a non-uniform prior Most likely coin: Best estimate for P(H) C 3 P(H|C 3 ) = 0.9 C 3 P(H|C 3 ) = 0.9 P(C 3 ) = 0.70 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 20 / 45

Lecture 2 Introduction to probabilistic inference Did We Do The Right Thing? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 21 / 45

Lecture 2 Introduction to probabilistic inference Did We Do The Right Thing? P(C 1 |HT) = 0.035 P(C 2 |HT) = 0.481 P(C 3 |HT) = 0.485 C 2 and C 3 are almost equally likely C 1 C 2 C 3 P(H|C 1 ) = 0.1 P(H|C 3 ) = 0.9 P(H|C 2 ) = 0.5 (Slide credit: University of Washington CSE473) S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 22 / 45

Information Theory and Statistical Inference Samuel Cheng School of - PowerPoint PPT Presentation

Information Theory and Statistical Inference Samuel Cheng School of ECE University of Oklahoma August 23, 2018 S. Cheng (OU-ECE) Information Theory and Statistical Inference August 23, 2018 1 / 45 Lecture 2 Introduction to probabilistic

Chapter 1: Introduction to Information Theory Book: Information Theory, Inference, and

Chapter 8: Hypothesis Testing STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 4: Multiple Random Variables STK4011/9011: Statistical Inference Theory Johan Pensar

Chapter 2: Transformations and Expectations (a recap) STK4011/9011: Statistical Inference Theory

Review of Estimation Theory Berlin 2003 References: 1. X. Huang et. al., Spoken Language

UQ, STAT2201, 2017, Lecture 6 Unit 6 Statistical Inference Ideas. 1 Statistical Inference is

Theory of statistical inference: a lazy approach to obtaining asymptotic results in parametric

Chapter 3: Common Families of Distributions STK4011/9011: Statistical Inference Theory Johan

Foundations for Inference I Dajiang Liu @PHS525 Feb-09-2016 Statistical Inference

Statistical Natural Language Processing A refresher on information theory ar ltekin

Theory of Statistical Inference Dajiang Liu @PHS 525 Feb-11, 2016 Sampling Distribution for

Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Hypothesis testing and statistical decision theory Lirong Xia March 25, 2016 Schedule

Hypothesis testing and statistical decision theory Lirong Xia Fall, 2016 Schedule

Introduction to information theory and empirical statistical theory Nan Chen and Andrew J. Majda

Statistical Inference Fall 2018 Tyson S. Barrett, PhD The Whole Idea All can be done in R Why

Algorithms and Limits in Statistical Inference Jayadev Acharya Massachusetts Institute of

The Theory of Quantum Statistical Comparison and Some Applications in Quantum Information

II.2 Statistical Inference: Sampling and Estimation A statistical model is a set of

Samples and Statistics The objective of statistical inference is to draw conclusions or make

Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson

Modes of Statistical Inference for Causal Efgects Plus an overview of the testing based approach

The Theory of Statistical Comparison in Quantum Information and Foundations Francesco Buscemi *