CS70: Jean Walrand: Lecture 21. Events, Conditional Probability 1. - PowerPoint PPT Presentation

CS70: Jean Walrand: Lecture 21. Events, Conditional Probability 1. Probability Basics Review 2. Events 3. Conditional Probability

Probability Basics Review Setup: ◮ Random Experiment. Flip a fair coin twice. ◮ Probability Space. ◮ Sample Space: Set of outcomes, Ω . Ω = { HH , HT , TH , TT } (Note: Not Ω = { H , T } with two picks!) ◮ Probability: Pr [ ω ] for all ω ∈ Ω . Pr [ HH ] = ··· = Pr [ TT ] = 1 / 4 1. 0 ≤ Pr [ ω ] ≤ 1 . 2. ∑ ω ∈ Ω Pr [ ω ] = 1 .

Set notation review Ω Ω Ω A [ B A \ B A B Figure: Difference ( A , Figure: Union (or) Figure: Two events not B ) Ω Ω Ω A ∩ B A ∆ B ¯ A Figure: Complement Figure: Symmetric Figure: Intersection (not) difference (only one) (and)

Probability of exactly one ‘heads’ in two coin flips? Idea: Sum the probabilities of all the different outcomes that have exactly one ‘heads’: HT , TH . This leads to a definition! Definition: ◮ An event, E , is a subset of outcomes: E ⊂ Ω . ◮ The probability of E is defined as Pr [ E ] = ∑ ω ∈ E Pr [ ω ] .

Event: Example Ω Pr [ ω ] 3/10 Red Green 4/10 Yellow 2/10 Blue 1/10 Physical experiment Probability model Ω = { Red, Green, Yellow, Blue } Pr [ Red ] = 3 10 , Pr [ Green ] = 4 10 , etc. E = { Red , Green } ⇒ Pr [ E ] = 3 + 4 = 3 10 + 4 10 = Pr [ Red ]+ Pr [ Green ] . 10

Probability of exactly one heads in two coin flips? Sample Space, Ω = { HH , HT , TH , TT } . Uniform probability space: Pr [ HH ] = Pr [ HT ] = Pr [ TH ] = Pr [ TT ] = 1 4 . Event, E , “exactly one heads”: { TH , HT } . Pr [ ω ] = | E | | Ω | = 2 4 = 1 Pr [ E ] = ∑ 2 . ω ∈ E

Example: 20 coin tosses. 20 coin tosses Sample space: Ω = set of 20 fair coin tosses. Ω = { T , H } 20 ≡ { 0 , 1 } 20 ; | Ω | = 2 20 . ◮ What is more likely? ◮ ω 1 := ( 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 ) , or ◮ ω 2 := ( 1 , 0 , 1 , 1 , 0 , 0 , 0 , 1 , 0 , 1 , 0 , 1 , 1 , 0 , 1 , 1 , 1 , 0 , 0 , 0 )? 1 Answer: Both are equally likely: Pr [ ω 1 ] = Pr [ ω 2 ] = | Ω | . ◮ What is more likely? ( E 1 ) Twenty Hs out of twenty, or ( E 2 ) Ten Hs out of twenty? Answer: Ten Hs out of twenty. Why? There are many sequences of 20 tosses with ten Hs; | Ω | ≪ Pr [ E 2 ] = | E 2 | 1 only one with twenty Hs. ⇒ Pr [ E 1 ] = | Ω | . � 20 � | E 2 | = = 184 , 756 . 10

Probability of n heads in 100 coin tosses. Ω = { H , T } 100 ; | Ω | = 2 100 . � 100 � Event E n = ‘ n heads’; | E n | = n | Ω | = ( 100 n ) p n := Pr [ E n ] = | E n | 2 100 p n Observe: ◮ Concentration around mean: Law of Large Numbers; ◮ Bell-shape: Central Limit n Theorem.

Roll a red and a blue die.

Exactly 50 heads in 100 coin tosses. Sample space: Ω = set of 100 coin tosses = { H , T } 100 . | Ω | = 2 × 2 ×···× 2 = 2 100 . 1 Uniform probability space: Pr [ ω ] = 2 100 . Event E = “100 coin tosses with exactly 50 heads” | E | ? Choose 50 positions out of 100 to be heads. � 100 � | E | = . 50 � 100 � 50 Pr [ E ] = 2 100 .

Calculation. Stirling formula (for large n ): √ � n � n n ! ≈ 2 π n . e √ 4 π n ( 2 n / e ) 2 n 4 n � 2 n � ≈ √ 2 π n ( n / e ) n ] 2 ≈ √ π n . n [ Pr [ E ] = | E | | Ω | = | E | 1 1 2 2 n = √ π n = √ ≈ . 08 . 50 π

Exactly 50 heads in 100 coin tosses.

Probability is Additive Theorem (a) If events A and B are disjoint, i.e., A ∩ B = / 0 , then Pr [ A ∪ B ] = Pr [ A ]+ Pr [ B ] . (b) If events A 1 ,..., A n are pairwise disjoint, i.e., A k ∩ A m = / 0 , ∀ k � = m , then Pr [ A 1 ∪···∪ A n ] = Pr [ A 1 ]+ ··· + Pr [ A n ] . Proof: Obvious.

Consequences of Additivity Theorem (a) Pr [ A ∪ B ] = Pr [ A ]+ Pr [ B ] − Pr [ A ∩ B ] ; (inclusion-exclusion property) (b) Pr [ A 1 ∪···∪ A n ] ≤ Pr [ A 1 ]+ ··· + Pr [ A n ] ; (union bound) (c) If A 1 ,... A N are a partition of Ω , i.e., pairwise disjoint and ∪ N m = 1 A m = Ω , then Pr [ B ] = Pr [ B ∩ A 1 ]+ ··· + Pr [ B ∩ A N ] . (law of total probability) Proof: (b) is obvious. See next two slides for (a) and (c).

Inclusion/Exclusion Pr [ A ∪ B ] = Pr [ A ]+ Pr [ B ] − Pr [ A ∩ B ]

Total probability Assume that Ω is the union of the disjoint sets A 1 ,..., A N . Then, Pr [ B ] = Pr [ A 1 ∩ B ]+ ··· + Pr [ A N ∩ B ] . Indeed, B is the union of the disjoint sets A n ∩ B for n = 1 ,..., N .

Roll a Red and a Blue Die. E 1 = ‘Red die shows 6’ ; E 2 = ‘Blue die shows 6’ E 1 ∪ E 2 = ‘At least one die shows 6’ Pr [ E 1 ] = 6 36 , Pr [ E 2 ] = 6 36 , Pr [ E 1 ∪ E 2 ] = 11 36 .

Conditional probability: example. Two coin flips. First flip is heads. Probability of two heads? Ω = { HH , HT , TH , TT } ; Uniform probability space. Event A = first flip is heads: A = { HH , HT } . New sample space: A ; uniform still. Event B = two heads. The probability of two heads if the first flip is heads. The probability of B given A is 1 / 2.

A similar example. Two coin flips. At least one of the flips is heads. → Probability of two heads? Ω = { HH , HT , TH , TT } ; uniform. Event A = at least one flip is heads. A = { HH , HT , TH } . New sample space: A ; uniform still. Event B = two heads. The probability of two heads if at least one flip is heads. The probability of B given A is 1 / 3.

Conditional Probability: A non-uniform example Ω Pr [ ω ] 3/10 Red Green 4/10 Yellow 2/10 Blue 1/10 Physical experiment Probability model Ω = { Red, Green, Yellow, Blue } Pr [ Red | Red or Green ] = 3 7 = Pr [ Red ∩ (Red or Green) ] Pr [ Red or Green ]

Another non-uniform example Consider Ω = { 1 , 2 ,..., N } with Pr [ n ] = p n . Let A = { 3 , 4 } , B = { 1 , 2 , 3 } . p 3 = Pr [ A ∩ B ] Pr [ A | B ] = . p 1 + p 2 + p 3 Pr [ B ]

Yet another non-uniform example Consider Ω = { 1 , 2 ,..., N } with Pr [ n ] = p n . Let A = { 2 , 3 , 4 } , B = { 1 , 2 , 3 } . p 2 + p 3 = Pr [ A ∩ B ] Pr [ A | B ] = . p 1 + p 2 + p 3 Pr [ B ]

Conditional Probability. Definition: The conditional probability of B given A is Pr [ B | A ] = Pr [ A ∩ B ] Pr [ A ] A ∩ B In A ! A A B B In B ? Must be in A ∩ B . Pr [ B | A ] = Pr [ A ∩ B ] Pr [ A ] .

Summary Events, Conditional Probability Key Ideas: ◮ Conditional Probability: Pr [ A | B ] = Pr [ A ∩ B ] Pr [ B ] ◮ All these are possible: Pr [ A | B ] < Pr [ A ]; Pr [ A | B ] > Pr [ A ]; Pr [ A | B ] = Pr [ A ] .