Preliminaries 32 / 384 Random variables Let V = { V 1 , . . . , V n - PowerPoint PPT Presentation

Chapter 2: Preliminaries 32 / 384

Random variables Let V = { V 1 , . . . , V n } , n ≥ 1 , be a set of random variables. Each variable V i ∈ V can take on one of m ≥ 2 values; for now we consider 2-valued variables: • V i = true , denoted by v i ; • V i = false , denoted by ¬ v i (or by v i ). The set V spans a Boolean Algebra of logical propositions V : • T(rue), F(alse) ∈ V ; • for all variables V i ∈ V we have that v i ∈ V ; • for all x ∈ V we have that ¬ x ∈ V ; • for all x, y ∈ V we have that x ∧ y ∈ V and x ∨ y ∈ V . The elements of V obey the usual rules of propositional logic. 33 / 384

The joint probability distribution Definition : Let V be the Boolean Algebra of propositions spanned by a set of random variables V . Let Pr : V → [0 , 1] be a function such that • Pr is positive: for each x ∈ V we have that Pr( x ) ≥ 0 and, more specifically, Pr( F ) = 0 ; • Pr is normed: Pr( T ) = 1 ; • Pr is additive: we have, for each x, y ∈ V with x ∧ y ≡ F, that Pr( x ∨ y ) = Pr( x ) + Pr( y ) . The function Pr is a joint probability distribution on V ; the function value Pr( x ) is the probability of x . 34 / 384

Independence of propositions Definition : Let V be the Boolean Algebra of propositions spanned by a set of random variables V . Let Pr be a joint probability distribution on V . Two propositions x, y ∈ V are called independent in Pr if Pr( x ∧ y ) = Pr( x ) · Pr( y ) The propositions x, y ∈ V are called conditionally independent given the proposition z ∈ V if we have that Pr( x ∧ y | z ) = Pr( x | z ) · Pr( y | z ) 35 / 384

The two notions of independence (1) • Consider two propositions x, y ∈ V such that x and y are independent 1 : x y Can z ∈ V exist such that x and y are dependent given z ? x • Yes: z y 1 The square has area 1, representing the total probability mass. 36 / 384

The two notions of independence (2) • Consider two propositions x, y ∈ V such that x and y are dependent: x y Can z ∈ V exist such that x and y are conditionally independent given z ? x • Yes: z y 37 / 384

Configurations Let V be a set of random variables and let W ⊆ V . • a configuration c W of W is a conjunction of value assignments to the variables from W ; • convention: c ∅ = T; • w is used to denote a specific configuration of W . • W also indicates all possible configurations to the set W (notation abuse!): W is then considered to be a template that can be filled in with any configuration c W . Example : Let W = { V 1 , V 3 , V 7 } . W = V 1 ∧ V 3 ∧ V 7 denotes a configuration template: filling in values for V i results in proper propositions/configurations. Some configurations c W of W are: ∧ ∧ V 1 = true V 3 = true V 7 = false v 1 ∧ ¬ v 3 ∧ v 7 ¬ v 1 ∧ ∧ ¬ v 7 v 3 � 38 / 384

Conventions and notation In the remainder of this course, for distributions on V : • rather than talking about propositions x ∈ V spanned by V we refer to configurations c V of V Set (bold faced) Singleton V Variables/templates (capital) V Values/configurations c V , v c V , v • conjunctions are often left implicit: e.g. v 1 v 2 denotes v 1 ∧ v 2 ; • note the following differences (!) Pr( c V ) , Pr( c V ) , Pr( v ) , Pr( v ) , Pr( v | c E ) probabilities: distributions: Pr( V ) , Pr( V ) , Pr( V | e ) Pr( V | E ) , Pr( V | E ) distribution sets : 39 / 384

Independence of variables Definition : Let V be a set of random variables and let X , Y , Z ⊆ V . Let Pr be a joint distribution on V . The set of variables X is called conditionally independent of the set Y given the set Z in Pr , if we have that Pr( X | Y ∧ Z ) = Pr( X | Z ) Remarks : • the expression Pr( X | Y ∧ Z ) = Pr( X | Z ) represents that Pr( c X | c Y ∧ c Z ) = Pr( c X | c Z ) holds for all configurations c X , c Y and c Z of X , Y and Z ; • Pr( X | Y ∧ Z ) = Pr( X | Z ) ⇒ Pr( X ∧ Y | Z ) = Pr( X | Z ) · Pr( Y | Z ) (what about ⇐ ?). � 40 / 384

Chapter 3: Independences and Graphical Representations 41 / 384

A qualitative notion of independence Observation : People are capable of making statements about independences among variables without having to perform numerical calculations. Conclusion : In human reasoning behaviour, the qualitative notion of independence is more fundamental than the quantitative notion of independence. 43 / 384

The (probabilistic) independence relation of a joint distribution Definition : Let V be a set of random variables and let Pr be a joint probability distribution on V . The independence relation I Pr of Pr is a set I Pr ⊆ P ( V ) × P ( V ) × P ( V ) , defined for all X , Y , Z ⊆ V by ( X , Z , Y ) ∈ I Pr if and only if Pr( X | Y ∧ Z ) = Pr( X | Z ) Remarks : • ( X , Z , Y ) ∈ I Pr will be written as I Pr ( X , Z , Y ) ; ( X , Z , Y ) / ∈ I Pr will be written as ¬ I Pr ( X , Z , Y ) ; • a statement I Pr ( X , Z , Y ) is called an independence statement for the joint distribution Pr . 44 / 384

Properties of I Pr : symmetry Lemma : I Pr ( X , Z , Y ) if and only if I Pr ( Y , Z , X ) Proof : I Pr ( X , Z , Y ) ⇐ ⇒ Pr( X | Y ∧ Z ) = Pr( X | Z ) ⇒ Pr( X ∧ Y ∧ Z ) = Pr( X ∧ Z ) ⇐ Pr( Y ∧ Z ) Pr( Z ) ⇒ Pr( X ∧ Y ∧ Z ) = Pr( Y ∧ Z ) ⇐ Pr( X ∧ Z ) Pr( Z ) ⇐ ⇒ Pr( Y | X ∧ Z ) = Pr( Y | Z ) ⇐ ⇒ I Pr ( Y , Z , X ) � 45 / 384

Properties of I Pr : decomposition Lemma : I Pr ( X , Z , Y ∪ W ) ⇒ I Pr ( X , Z , Y ) ∧ I Pr ( X , Z , W ) Proof : (sketch) (Note: c Y ∪ W = c Y ∧ c W !) Suppose that Pr( X | Y ∧ W ∧ Z ) = Pr( X | Z ) . Then, by definition, Pr( X ∧ Y ∧ W ∧ Z ) = Pr( Y ∧ W ∧ Z ) · Pr( X ∧ Z ) Pr( Z ) For Pr( X | Y ∧ Z ) we find that Pr( X | Y ∧ Z ) = Pr( X ∧ Y ∧ Z ) Pr( Y ∧ Z ) � c W Pr( X ∧ Y ∧ Z ∧ c W ) = Pr( Y ∧ Z ) = Pr( X ∧ Z ) = Pr( X | Z ) � Pr( Z ) 46 / 384

Properties of I Pr : weak union, contraction Lemma : • if I Pr ( X , Z , Y ∪ W ) then I Pr ( X , Z ∪ W , Y ) (weak union); • if I Pr ( X , Z , W ) and I Pr ( X , Z ∪ W , Y ) then I Pr ( X , Z , Y ∪ W ) (contraction) • (for strictly positive Pr also the intersection property holds; see syllabus) Proof : left as exercise 3.1. What about ⇐ ? 47 / 384

The definition of the independence relation Joint Distribution Pr Independence Independence relation relation I Pr I Properties: Axioms: symmetry, symmetry, decomposition, decomposition, weak union, weak union, contraction contraction 48 / 384

The (qualitative) independence relation I Definition : Let V be a set of random variables and let X , Y , Z , W ⊆ V . An independence relation I on V is a ternary relation I ⊆ P ( V ) × P ( V ) × P ( V ) that satisfies the following properties: • if I ( X , Z , Y ) then I ( Y , Z , X ) ; • if I ( X , Z , Y ∪ W ) then I ( X , Z , Y ) and I ( X , Z , W ) ; • if I ( X , Z , Y ∪ W ) then I ( X , Z ∪ W , Y ) ; • if I ( X , Z , W ) and I ( X , Z ∪ W , Y ) then I ( X , Z , Y ∪ W ) . The first property is called the symmetry axiom ; the second is called the decomposition axiom; the third is referred to as the weak union axiom; the last one is called contraction. 49 / 384

An example Lemma : Let I be an independence relation on a set of random variables V . We have that if I ( X , Z , Y ) and I ( X ∪ Z , Y , W ) then I ( X , Z , W ) for all X , Y , Z , W ⊆ V . Proof : We observe that I ( X ∪ Z , Y , W ) ⇒ symm I ( W , Y , X ∪ Z ) ⇒ weakunion ⇒ I ( W , Y ∪ Z , X ) ⇒ symm I ( X , Y ∪ Z , W ) From I ( X , Z , Y ) , I ( X , Y ∪ Z , W ) and the contraction axiom we have that I ( X , Z , W ∪ Y ) ; decomposition now gives I ( X , Z , W ) . � 50 / 384

Representing independences Different ways exist of representing an independence relation: • all independence statements of the relation are explicitly stated; • only the independence statements of a suitable subset of the relation are explicitly stated — all other statements are implicitly represented by means of the axioms; • the independence relation is coded in a graph; • . . . 51 / 384