Probability Review Gonzalo Mateos Dept. of ECE and Goergen - PowerPoint PPT Presentation

Probability Review Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ August 30, 2020 Introduction to Random Processes Probability Review 1

Sigma-algebras and probability spaces Sigma-algebras and probability spaces Conditional probability, total probability, Bayes’ rule Independence Random variables Discrete random variables Continuous random variables Expected values Joint probability distributions Joint expectations Introduction to Random Processes Probability Review 2

Probability ◮ An event is something that happens ◮ A random event has an uncertain outcome ⇒ The probability of an event measures how likely it is to occur Example ◮ I’ve written a student’s name in a piece of paper. Who is she/he? ◮ Event: Student x ’s name is written in the paper ◮ Probability: P ( x ) measures how likely it is that x ’s name was written ◮ Probability is a measurement tool ⇒ Mathematical language for quantifying uncertainty Introduction to Random Processes Probability Review 3

Sigma-algebra ◮ Given a sample space or universe S ◮ Ex: All students in the class S = { x 1 , x 2 , . . . , x N } ( x n denote names) ◮ Def: An outcome is an element or point in S , e.g., x 3 ◮ Def: An event E is a subset of S ◮ Ex: { x 1 } , student with name x 1 ◮ Ex: Also { x 1 , x 4 } , students with names x 1 and x 4 ⇒ Outcome x 3 and event { x 3 } are different, the latter is a set ◮ Def: A sigma-algebra F is a collection of events E ⊆ S such that (i) The empty set ∅ belongs to F : ∅ ∈ F (ii) Closed under complement: If E ∈ F , then E c ∈ F (iii) Closed under countable unions: If E 1 , E 2 , . . . ∈ F , then ∪ ∞ i =1 E i ∈ F ◮ F is a set of sets Introduction to Random Processes Probability Review 4

Examples of sigma-algebras Example ◮ No student and all students, i.e., F 0 := {∅ , S } Example ◮ Empty set, women, men, everyone, i.e., F 1 := {∅ , Women , Men , S } Example ◮ F 2 including the empty set ∅ plus All events (sets) with one student { x 1 } , . . . , { x N } plus All events with two students { x 1 , x 2 } , { x 1 , x 3 } , . . . , { x 1 , x N } , { x 2 , x 3 } , . . . , { x 2 , x N } , . . . { x N − 1 , x N } plus All events with three, four, . . . , N students ⇒ F 2 is known as the power set of S , denoted 2 S Introduction to Random Processes Probability Review 5

Axioms of probability ◮ Define a function P ( E ) from a sigma-algebra F to the real numbers ◮ P ( E ) qualifies as a probability if A1) Non-negativity: P ( E ) ≥ 0 A2) Probability of universe: P ( S ) = 1 A3) Additivity: Given sequence of disjoint events E 1 , E 2 , . . . � ∞ � ∞ � � P E i = P ( E i ) i =1 i =1 ⇒ Disjoint (mutually exclusive) events means E i ∩ E j = ∅ , i � = j ⇒ Union of countably infinite many disjoint events ◮ Triplet ( S , F , P ( · )) is called a probability space Introduction to Random Processes Probability Review 6

Consequences of the axioms ◮ Implications of the axioms A1)-A3) ⇒ Impossible event: P ( ∅ ) = 0 ⇒ Monotonicity: E 1 ⊂ E 2 ⇒ P ( E 1 ) ≤ P ( E 2 ) ⇒ Range: 0 ≤ P ( E ) ≤ 1 ⇒ Complement: P ( E c ) = 1 − P ( E ) ⇒ Finite disjoint union: For disjoint events E 1 , . . . , E N � N � N � � P E i = P ( E i ) i =1 i =1 ⇒ Inclusion-exclusion: For any events E 1 and E 2 P ( E 1 ∪ E 2 ) = P ( E 1 ) + P ( E 2 ) − P ( E 1 ∩ E 2 ) Introduction to Random Processes Probability Review 7

Probability example ◮ Let’s construct a probability space for our running example ◮ Universe of all students in the class S = { x 1 , x 2 , . . . , x N } ◮ Sigma-algebra with all combinations of students, i.e., F = 2 S ◮ Suppose names are equiprobable ⇒ P ( { x n } ) = 1 / N for all n ⇒ Have to specify probability for all E ∈ F ⇒ Define P ( E ) = | E | | S | ◮ Q: Is this function a probability? ⇒ A1): P ( E ) = | E | | S | ≥ 0 � ⇒ A2): P ( S ) = | S | | S | = 1 � = | � N i =1 E i | � N �� N � i =1 | E i | = � N ⇒ A3): P i =1 E i = i =1 P ( E i ) � | S | | S | ◮ The P ( · ) just defined is called uniform probability distribution Introduction to Random Processes Probability Review 8

Conditional probability, total probability, Bayes’ rule Sigma-algebras and probability spaces Conditional probability, total probability, Bayes’ rule Independence Random variables Discrete random variables Continuous random variables Expected values Joint probability distributions Joint expectations Introduction to Random Processes Probability Review 9

Conditional probability ◮ Consider events E and F , and suppose we know F occurred ◮ Q: What does this information imply about the probability of E ? ◮ Def: Conditional probability of E given F is (need P ( F ) > 0) � F ) = P ( E ∩ F ) � P ( E P ( F ) ⇒ In general P ( E | F ) � = P ( F | E ) F ◮ Renormalize probabilities to the set F E 2 ∩ F E 2 ◮ Discard a piece of S ◮ May discard a piece of E as well E 1 S ◮ For given F with P ( F ) > 0, P ( ·| F ) satisfies the axioms of probability Introduction to Random Processes Probability Review 10

Conditional probability example ◮ The name I wrote is male. What is the probability of name x n ? ◮ Assume male names are F = { x 1 , . . . , x M } ⇒ P ( F ) = M N ◮ If name x n is male, x n ∈ F and we have for event E = { x n } P ( E ∩ F ) = P ( { x n } ) = 1 N ⇒ Conditional probability is as you would expect � F ) = P ( E ∩ F ) = 1 / N M / N = 1 � P ( E P ( F ) M ◮ If name is female x n / ∈ F , then P ( E ∩ F ) = P ( ∅ ) = 0 � � F ) = 0 ⇒ As you would expect, then P ( E Introduction to Random Processes Probability Review 11

Law of total probability ◮ Consider event E and events F and F c ◮ F and F c form a partition of the space S ( F ∪ F c = S , F ∩ F c = ∅ ) ◮ Because F ∪ F c = S cover space S , can write the set E as E = E ∩ S = E ∩ [ F ∪ F c ] = [ E ∩ F ] ∪ [ E ∩ F c ] ◮ Because F ∩ F c = ∅ are disjoint, so is [ E ∩ F ] ∩ [ E ∩ F c ] = ∅ ⇒ P ( E ) = P ([ E ∩ F ] ∪ [ E ∩ F c ]) = P ( E ∩ F ) + P ( E ∩ F c ) ◮ Use definition of conditional probability � � � F ) P ( F ) + P ( E � F c ) P ( F c ) P ( E ) = P ( E � � � F ) and P ( E � F c ) ◮ Translate conditional information P ( E ⇒ Into unconditional information P ( E ) Introduction to Random Processes Probability Review 12

Law of total probability (continued) F 1 F 3 ◮ In general, consider (possibly infinite) partition F i , i = 1 , 2 , . . . of S E ∩ F 3 E ∩ F 1 E ∩ F 2 ◮ Sets are disjoint ⇒ F i ∩ F j = ∅ for i � = j ◮ Sets cover the space ⇒ ∪ ∞ i =1 F i = S F 2 ◮ As before, because ∪ ∞ i =1 F i = S cover the space, can write set E as � ∞ � ∞ � � E = E ∩ S = E ∩ F i = [ E ∩ F i ] i =1 i =1 ◮ Because F i ∩ F j = ∅ are disjoint, so is [ E ∩ F i ] ∩ [ E ∩ F j ] = ∅ . Thus � ∞ � ∞ ∞ � � � � � F i ) P ( F i ) [ E ∩ F i ] P ( E ∩ F i ) = P ( E ) = P = P ( E i =1 i =1 i =1 Introduction to Random Processes Probability Review 13

Total probability example ◮ Consider a probability class in some university ⇒ Seniors get an A with probability (w.p.) 0.9, juniors w.p. 0.8 ⇒ An exchange student is a senior w.p. 0.7, and a junior w.p. 0.3 ◮ Q: What is the probability of the exchange student scoring an A? ◮ Let A = “exchange student gets an A,” S denote senior, and J junior ⇒ Use the law of total probability � � � S ) P ( S ) + P ( A � J ) P ( J ) P ( A ) = P ( A = 0 . 9 × 0 . 7 + 0 . 8 × 0 . 3 = 0 . 87 Introduction to Random Processes Probability Review 14

Bayes’ rule ◮ From the definition of conditional probability � � F ) P ( F ) = P ( E ∩ F ) P ( E ◮ Likewise, for F conditioned on E we have � � E ) P ( E ) = P ( F ∩ E ) P ( F ◮ Quantities above are equal, giving Bayes’ rule � � E ) P ( E ) � F ) = P ( F � P ( E P ( F ) ◮ Bayes’ rule allows time reversion. If F (future) comes after E (past), � � F ), probability of past ( E ) having seen the future ( F ) ⇒ P ( E � � E ), probability of future ( F ) having seen past ( E ) ⇒ P ( F � � � past. Interest is often in past � future ◮ Models often describe future Introduction to Random Processes Probability Review 15

Bayes’ rule example ◮ Consider the following partition of my email ⇒ E 1 =“spam” w.p. P ( E 1 ) = 0 . 7 ⇒ E 2 =“low priority” w.p. P ( E 2 ) = 0 . 2 ⇒ E 3 =“high priority” w.p. P ( E 3 ) = 0 . 1 ◮ Let F =“an email contains the word free ” � � E 1 ) = 0 . 9, P ( F � E 2 ) = P ( F � � � E 3 ) = 0 . 01 ⇒ From experience know P ( F ◮ I got an email containing “free”. What is the probability that it is spam? ◮ Apply Bayes’ rule � E 1 ) P ( E 1 ) � E 1 ) P ( E 1 ) � � � F ) = P ( F P ( F � P ( E 1 = = 0 . 995 � 3 � E i ) P ( E i ) P ( F ) � i =1 P ( F ⇒ Law of total probability very useful when applying Bayes’ rule Introduction to Random Processes Probability Review 16

Independence Sigma-algebras and probability spaces Conditional probability, total probability, Bayes’ rule Independence Random variables Discrete random variables Continuous random variables Expected values Joint probability distributions Joint expectations Introduction to Random Processes Probability Review 17