Simplicity and Complexity of Belief-Propagation Elchanan Mossel 1 1 - PowerPoint PPT Presentation

Simplicity and Complexity of Belief-Propagation Elchanan Mossel 1 1 MIT Dec 2020 Elchanan Mossel Simplicity & Complexity of BP

Markov Random Fields and Information Flow on Trees Consider the following process on a tree. Color the root randomly. Repeat: Copy color of parent with probability θ . Otherwise, chose color ∼ U [ q ]. Will mostly consider full d -ary tree / Branching process trees. Elchanan Mossel Simplicity & Complexity of BP

Inference - Machine Learning Perspective Inference problem (V1): Infer root color from leaf colors? A: No! Even in one level, ∃ randomness in root given children. Inference Problem (V2): How much can infer on the root color from leaf colors? Machine Learning: We can compute the posterior exactly! Moreover: Belief-Propagation does it in linear time. Remark: Belief Propagation is often applied to non-tree graphs [Pearl 82]. Applications to trees where known in biology and statistical physics before [Hidden 1970, Preston 1974]. Elchanan Mossel Simplicity & Complexity of BP

Part 1 : q = 2 - linear theory Part 1: LINEAR THEORY q = 2 Elchanan Mossel Simplicity & Complexity of BP

Inference - Asymptotic Perspective Let q = 2, fix the d -ary tree of h levels and call the two colors +1 , − 1. Let X v be the color of node v . Let X 0 denote the root color and X h are labels at level h of the tree. Inference Problem (V2): How much can infer on the root color from leaf colors? Q1: Can we analyze the optimal estimator (BP)? Q2: Is Majority = sgn ( � i X h ( i )) a good estimator as h → ∞ ? Question asked in Statistical Physics in terms of the extremality of the free measure of the Ising/Potts model on the Bethe lattice. Elchanan Mossel Simplicity & Complexity of BP

The Majority Estimator Let S h = � i X h ( i ). Exercise 1. E [ S h | X 0 ] = ( d θ ) h X 0 and � C ( θ ) , d θ 2 > 1 , E [ S h | X 0 ] 2 ( d θ ) 2 h � Var [ S h | X 0 ] = Var [ S h | X 0 ] → d θ 2 ≤ 1 0 , ⇒ lim h →∞ d TV ( S h | X 0 = + , S h | X 0 = − ) > 0 if d θ 2 > 1. = ⇒ lim h →∞ E [ sgn ( S h ) X 0 ] > 0 if d θ 2 > 1. = Analyzing Fourier Transform of S h Kesten-Stigum-66 proved: d θ 2 ≤ 1 = ⇒ S h → h →∞ a normal law independent of X 0 ( ∗ ). d θ 2 > 1 = ⇒ S h → h →∞ a non-normal law dependent on X 0 . d θ 2 = 1 is referred to as the Kesten-Stigum threshold. Exercise: Apply the martingale CLT, to prove the normal case. Elchanan Mossel Simplicity & Complexity of BP

Perspectives United: Thm for q = 2: If d θ 2 ≤ 1 then P [ BP → h →∞ (0 . 5 , 0 . 5)] = 1. = ⇒ BP infers non-trivially iff Majority infers non-trivially. Multiple proofs: Bleher, Ruiz, and Zagrebnov (95), Ioffe (96), Evans-Kenyon-Peres-Schulmann (00), Borg, Tour-Chayes, M, Roch (06), etc. (also: Chayes, Chayes, , Sethna, Thouless, (1986)). EKPS: Also for random trees, where d is the average degree. All use some concavity of the functionals of the distribution. Next: A proof sketch and applications areas of the linear theory. Elchanan Mossel Simplicity & Complexity of BP

Recursion of Random Variables for Binary Tree Let P + T denote the measure of X h when root is +, let P T = 0 . 5( P − T + P + T ). M := M T := P T [ X 0 = + | X h ] − P T [ X 0 = −| X h ]. Ex: (Bayes): dP ± E + T [ M ] = E T [ M 2 ] = E + T [ M 2 ]. dP T = 1 ± M , T Claim 1: If T = 0 − > S , M = M T , N = M S : M = θ N , E + T [ N ] = θ E + S [ N ] , E + T [ N 2 ] = θ E + S [ N 2 ] + (1 − θ ) E S [ N 2 ]. Claim 2:: If T 1 , T 2 are two trees joined at the root to form T and N i = X T i then: M = N 1 + N 2 . 1 + N 1 N 2 M n + M ′ n = ⇒ Belief Propagation Recursion : M n +1 = θ . 1 + θ 2 M n M ′ n Elchanan Mossel Simplicity & Complexity of BP

2 θ 2 < 1 = ⇒ E [ M 2 n ] → 0, i.e., asymptotic indpendence r 2 N 1 + N 2 1 M = θ , 1 + r = 1 − r + 1 + r = ⇒ 1 + θ 2 N 1 N 2 M = θ ( N 1 + N 2 ) − θ 3 N 1 N 2 ( N 1 + N 2 ) + θ 4 N 2 1 N 2 2 M M ≤ θ ( N 1 + N 2 ) − θ 3 N 1 N 2 ( N 1 + N 2 ) + θ 4 N 2 1 N 2 2 ⇒ (taking E + T recalling E + T [ M ] = E T [ M 2 ]) = E T [ M 2 ] ≤ 2 θ 2 E S [ N 2 ] − θ 4 E S + [ N 2 ] = ⇒ E [ M 2 n ] ≤ (2 θ 2 ) n Elchanan Mossel Simplicity & Complexity of BP

Application 1: The Phylogenetic Inference Problem 1 2 3 4 5 T T T A A A G G G C C C A A C G G C C C C C C T C C Elchanan Mossel Simplicity & Complexity of BP