CPCS Networks – Medical Diagnosis (noisy-OR model) Test case: no evidence Anytime-mpe(0.0001) U/L error vs time 3.8 cpcs422b 3.4 cpcs360b 3.0 Upper/Lower 2.6 2.2 1.8 1.4 1.0 0.6 i=1 i=21 1 10 100 1000 Time and parameter i Time (sec) Algorithm cpcs360 cpcs422 elim-mpe 115.8 1697.6 10 − = anytime-mpe( ), 70.3 505.2 4 10 − = 1 anytime-mpe( ), 70.3 110.5 slides10 828X 2019
Outline • Mini-bucket elimination • Weighted Mini-bucket • Mini-clustering • Re-parameterization, cost-shifting • Iterative Belief propagation • Iterative-join-graph propagation slides10 828X 2019
Decomposition for Sum • Generalize technique to sum via Holder’s inequality: • Define the weighted (or powered) sum: • “Temperature” interpolates between sum & max: • Different weights do not commute: slides10 828X 2019
The Power Sum and Holder Inequality slides10 828X 2019
Working Example • Model: • Markov network • Task: A • Partition function B C (Qiang Liu slides) slides10 828X 2019
Mini-Bucket (Basic Principles) • Upper bound • Lower bound (Qiang Liu slides) slides10 828X 2019
Holder Inequality • Where and • When , the equality is achieved. (Qiang Liu slides) G. H. Hardy, J. E. Littlewood and G. Pólya, Inequalities, Cambridge Univ. Press, London and New York, 1934. •
Reverse Holder Inequality • If , but the direction of the inequality reverses. (Qiang Liu slides) G. H. Hardy, J. E. Littlewood and G. Pólya, Inequalities, Cambridge Univ. Press, London and New York, 1934.
Weighted Mini-Bucket (for summation) 𝑥 Exact bucket elimination: mini-buckets ෑ 𝑔(𝑦) 𝑦 𝜇 𝐶 𝑏, 𝑑, 𝑒, 𝑓 = 𝑔 𝑏, 𝑐 ⋅ 𝑔 𝑐, 𝑑 ⋅ 𝑔 𝑐, 𝑒 ⋅ 𝑔 𝑐, 𝑓 𝑐 bucket B: 𝑔 𝑏, 𝑐 𝑔 𝑐, 𝑑 𝑔 𝑐, 𝑒 𝑔 𝑐, 𝑓 𝑥 1 𝑥 2 ≤ 𝑔 𝑏, 𝑐 𝑔 𝑐, 𝑑 ⋅ 𝑔 𝑐, 𝑒 𝑔 𝑐, 𝑓 𝑐 𝑐 bucket C: 𝜇 𝐶→𝐷 (𝑏, 𝑑) 𝑔 𝑏, 𝑑 𝑔 𝑑, 𝑓 = 𝜇 𝐶→𝐷 (𝑏, 𝑑) ⋅ 𝜇 𝐶→𝐸 (𝑒, 𝑓) 𝑔 𝑏, 𝑒 𝜇 𝐶→𝐸 (𝑒, 𝑓) (mini-buckets) bucket D: 𝑥 𝑥 1 𝜇 𝐷→𝐹 (𝑏, 𝑓) 𝜇 𝐸→𝐹 (𝑏, 𝑓) where bucket E: 𝑔 𝑦 = 𝑔 𝑦 𝑥 𝑦 𝑦 is the weighted or “power” sum operator 𝑔 𝑏 bucket A: 𝜇 𝐹→𝐵 (𝑏) 𝑥 𝑥 1 𝑥 2 𝑔 1 𝑦 𝑔 2 𝑦 ≤ 𝑔 1 𝑦 𝑔 2 𝑦 U = upper bound 𝑦 𝑦 𝑦 where 𝑥 1 + 𝑥 2 = 𝑥 and 𝑥 1 > 0, 𝑥 2 > 0 [Liu and Ihler, 2011] (lower bound if ) 𝑥 1 > 0, 𝑥 2 < 0 slides10 828X 2019
slides10 828X 2019
Weighted-mini-bucket for Marginal Map slides10 828X 2019
Bucket Elimination for MMAP Bucket Elimination A B: constrained elimination order SUM B C C: D E D: E: MAX A: MAP* is the marginal MAP value slides7 828X 2019
A MB and WMB for Marginal MAP B C D E mini-buckets Marginal MAP 𝑥 1 𝜇 𝐶→𝐷 𝑏, 𝑑 = 𝑔 𝑏, 𝑐 𝑔(𝑐, 𝑑) Σ 𝐶 bucket B: 𝑔 𝑏, 𝑐 𝑔 𝑐, 𝑑 𝑔 𝑐, 𝑒 𝑔 𝑐, 𝑓 𝑐 𝑥 2 𝜇 𝐶→𝐸 𝑒, 𝑓 = 𝑔 𝑐, 𝑒 𝑔(𝑐, 𝑓) Σ 𝐷 bucket C: 𝜇 𝐶→𝐷 (𝑏, 𝑑) 𝑔 𝑏, 𝑑 𝑔 𝑑, 𝑓 𝑐 (𝑥 1 + 𝑥 2 = 1) . . 𝑔 𝑏, 𝑒 𝜇 𝐶→𝐸 (𝑒, 𝑓) max D bucket D: . max E 𝜇 𝐷→𝐹 (𝑏, 𝑓) 𝜇 𝐸→𝐹 (𝑏, 𝑓) bucket E: 𝜇 𝐹→𝐵 𝑏 = max 𝜇 𝐷→𝐹 𝑏, 𝑓 𝜇 𝐸→𝐹 (𝑏, 𝑓) 𝑓 max A 𝑔 𝑏 bucket A: 𝜇 𝐹→𝐵 (𝑏) 𝑉 = max 𝑔 𝑏 𝜇 𝐹→𝐵 (𝑏) 𝑏 U = upper bound Can optimize over cost-shifting and weights (single pass “MM” or iterative message passing) [Liu and Ihler, 2011; 2013] [Dechter and Rish, 2003] slides10 828X 2019
MBE-map Process max buckets With max mini-buckets And sum buckets with weighted Mini-buckets slides10 828X 2019
Initial partitioning slides10 828X 2019
slides10 828X 2019
Complexity and Tractability of MBE(i,m) slides10 828X 2019
Outline • Mini-bucket elimination • Weighted Mini-bucket • Mini-clustering • Re-parameterization, cost-shifting • Iterative Belief propagation • Iterative-join-graph propagation slides10 828X 2019
Join-Tree Clustering (Cluster-Tree Elimination) A 1 ABC = h b c p a p b a p c a b ( , ) ( ) ( | ) ( | , ) ( 1 , 2 ) B a BC = h b c p d b p f c d h b f ( , ) ( | ) ( | , ) ( , ) ( 2 , 1 ) ( 3 , 2 ) d f , 2 BCDF C D E = h b f p d b p f c d h b c ( , ) ( | ) ( | , ) ( , ) ( 2 , 3 ) ( 1 , 2 ) c d , BF = F h b f p e b f h e f ( , ) ( | , ) ( , ) ( 3 , 2 ) ( 4 , 3 ) e 3 BEF = G h e f p e b f h b f ( , ) ( | , ) ( , ) EXACT algorithm ( 3 , 4 ) ( 2 , 3 ) b EF Time and space: = = h e f p G g e f ( , ) ( | , ) e ( 4 , 3 ) exp(cluster size)= 4 EFG exp(treewidth) slides10 828X 2019
We can replace the sum with power sum For weights that sum to 1 in each mini-bucket slides10 828X 2019
Mini-Clustering, i-bound=3 A A B C 1 B p(a), p(b|a), p(c|a,b) = 1 C D E h b c p a p b a p c a b ( , ) ( ) ( | ) ( | , ) BC ( 1 , 2 ) a F B C D G p(d|b), h (1,2) (b,c) 2 C D F = p(f|c,d) 1 1 h b p d b h b c ( ) ( | ) ( , ) ( 2 , 3 ) ( 1 , 2 ) c d , BF = 2 h f p f c d ( ) max ( | , ) ( 2 , 3 ) c d , B E F 3 p(e|b,f), h 1 (2,3) (b), h 2 (2,3) (f) APPROXIMATE algorithm EF Time and space: exp(i-bound) E F G 4 p(g|e,f) slides10 828X 2019 Number of variables in a mini-cluster
Mini-Clustering - Example A 1 ABC B = h 1 b c p a p b a p c a b ( , ) : ( ) ( | ) ( | , ) H ( 1 , 2 ) C D E ( 1 , 2 ) a = BC 1 1 h b p d b h b f ( ) : ( | ) ( , ) F ( 2 , 1 ) ( 3 , 2 ) d f , H = G h 2 c p f c d ( ) : ( | , ) max ( 2 , 1 ) ( 2 , 1 ) d f , 2 BCDF = 1 1 h b p d b h b c ( ) : ( | ) ( , ) ( 2 , 3 ) ( 1 , 2 ) H c d , ( 2 , 3 ) = h 2 f p f c d ( ) : max ( | , ) BF ( 2 , 3 ) c d , = h 1 b f p e b f h 1 e f H ( , ) : ( | , ) ( , ) ( 3 , 2 ) ( 4 , 3 ) ( 3 , 2 ) e 3 BEF = H h 1 e f p e b f h 1 b h 2 f ( , ) : ( | , ) ( ) ( ) ( 3 , 4 ) ( 3 , 4 ) ( 2 , 3 ) ( 2 , 3 ) EF b = = H h 1 e f p G g e f ( , ) : ( | , ) ( 4 , 3 ) e ( 4 , 3 ) 4 EFG slides10 828X 2019
A Cluster Tree Elimination vs. Mini-Clustering B C D E MC CTE F 1 1 ABC ABC h 1 b c h b c H ( , ) ( , ) G ( 1 , 2 ) ( 1 , 2 ) ( 1 , 2 ) 1 BC BC h b ( ) ( 2 , 1 ) h 2 c h b c H ( ) ( , ) ( 2 , 1 ) ( 2 , 1 ) ( 2 , 1 ) 2 2 BCDF BCDF h 1 b h b f ( ) ( , ) H ( 2 , 3 ) ( 2 , 3 ) ( 2 , 3 ) h 2 f ( ) BF BF ( 2 , 3 ) H h 1 b f ( , ) h b f ( , ) ( 3 , 2 ) ( 3 , 2 ) ( 3 , 2 ) 3 3 BEF BEF h e f H ( , ) h 1 e f ( , ) ( 3 , 4 ) ( 3 , 4 ) ( 3 , 4 ) EF EF h e f ( , ) H h 1 e f ( , ) ( 4 , 3 ) ( 4 , 3 ) ( 4 , 3 ) 4 4 EFG EFG slides10 828X 2019
Heuristics for partitioning (Dechter and Rish, 2003, Rollon and Dechter 2010) Scope-based Partitioning Heuristic (SCP) aims at minimizing the number of mini-buckets in the partition by including in each minibucket as many functions as respecting the i bound is satisfied Use greedy heuristic derived from a distance function to decide which functions go into a single mini-bucket slides10 828X 2019
Greedy Scope-based Partitioning slides10 828X 2019
Heuristic for Partitioning Scope-based Partitioning Heuristic. The scope-based partition heuristic (SCP) aims at minimizing the number of mini-buckets in the partition by including in each mini-bucket as many functions as possible as long as the i bound is satisfied. First, single function mini-buckets are decreasingly ordered according to their arity from left to right. Then, each mini-bucket is absorbed into the left-most mini-bucket with whom it can be merged. The time complexity of Partition( B, i ) , where B is the bucket to be partitioned, and |B|,the number of functions in the bucket, using the SCP heuristic is O ( |B| log ( |B| ) + |B|^ 2) . The scope-based heuristic is is quite fast, its shortcoming is that it does not consider the actual information in the functions. slides10 828X 2019
Greedy Partition as a function of a distance function h slides10 828X 2019
Comparing Mini-clustering against Belief Propagation. What is belief propagation slides10 828X 2019
Iterative Belief Proapagation • Belief propagation is exact for poly-trees • IBP - applying BP iteratively to cyclic networks U U U One step : 1 3 2 update 3 x 2 x ( ) ( ) U 1 U 1 BEL(U ) 1 1 u ( ) X 1 2 u ( ) X 1 X X 1 2 • No guarantees for convergence • Works well for many coding networks slides10 828X 2019
Linear Block Codes a b c d e f g h Received bits σ A B C D E F G H Input bits Gaussian channel noise + + + + + + Parity bits σ p 1 p 2 p 3 p 4 p 5 p 6 Received bits slides10 828X 2019
Probabilistic decoding Error-correcting linear block code State-of-the-art: approximate algorithm – iterative belief propagation (IBP) (Pearl’s poly -tree algorithm applied to loopy networks) slides10 828X 2019
MBE-mpe vs. IBP MBE-mpe is better on low w* codes IBP (or BP) is better on randomly generated (high w*) codes. Bit error rate (BER) as a function of noise (sigma): slides10 828X 2019
Grid 15x15 - 10 evidence Grid 15x15, evid=10, w*=22, 10 instances Grid 15x15, evid=10, w*=22, 10 instances 0.06 0.14 MC 0.12 0.05 IBP MC IBP 0.10 0.04 Absolute error 0.08 NHD 0.03 0.06 0.02 0.04 0.01 0.02 0.00 0.00 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 i-bound i-bound Grid 15x15, evid=10, w*=22, 10 instances Grid 15x15, evid=10, w*=22, 10 instances 0.12 12 MC MC 10 0.10 IBP IBP 8 0.08 Time (seconds) Relative error 6 0.06 4 0.04 2 0.02 0 0.00 slides10 828X 2019 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 i-bound i-bound
Outline • Mini-bucket elimination • Weighted Mini-bucket • Mini-clustering • Iterative Belief propagation • Iterative-join-graph propagation • Re-parameterization, cost-shifting slides10 828X 2019
Iterative Belief Proapagation • Belief propagation is exact for poly-trees • IBP - applying BP iteratively to cyclic networks U U U One step : 1 3 2 update 3 x 2 x ( ) ( ) U 1 U 1 BEL(U ) 1 1 u ( ) X 1 2 u ( ) X 1 X X 1 2 • No guarantees for convergence • Works well for many coding networks • Lets combine iterative-nature with anytime--IJGP slides10 828X 2019
Iterative Join Graph Propagation • Loopy Belief Propagation • Cyclic graphs • Iterative • Converges fast in practice (no guarantees though) • Very good approximations (e.g., turbo decoding, LDPC codes, SAT – survey propagation) • Mini-Clustering(i) • Tree decompositions • Only two sets of messages (inward, outward) • Anytime behavior – can improve with more time by increasing the i-bound • We want to combine: • Iterative virtues of Loopy BP • Anytime behavior of Mini-Clustering(i) slides10 828X 2019
IJGP - The basic idea • Apply Cluster Tree Elimination to any join-graph • We commit to graphs that are I-maps • Avoid cycles as long as I-mapness is not violated • Result: use minimal arc-labeled join-graphs slides10 828X 2019
Tree Decomposition for Belief Updating p(a ) A p(b|a ) B p(e b f | , ) P(d|b p(c a b ) E C D | , ) p(f|c , d ) F G p(g|e , f ) 93
Tree Decomposition for belief updating A B C p(a ) p(a), p(b|a), p(c|a,b) A BC p(b|a ) B B C D F p(d|b), p(f|c,d) p(e b f | , ) BF P(d|b p(c a b ) E C D | , ) B E F p(e|b,f) p(f|c , d ) EF F E F G p(g|e,f) G p(g|e , f ) 94
CT CTE: : Clu luster Tree Eli limination 1 ABC = A h b c p a p b a p c a b ( , ) ( ) ( | ) ( | , ) ( 1 , 2 ) a BC B = h b c p d b p f c d h b f ( , ) ( | ) ( | , ) ( , ) ( 2 , 1 ) ( 3 , 2 ) d f , 2 BCDF = h b f p d b p f c d h b c ( , ) ( | ) ( | , ) ( , ) ( 2 , 3 ) ( 1 , 2 ) c d , C D E BF = h b f p e b f h e f ( , ) ( | , ) ( , ) ( 3 , 2 ) ( 4 , 3 ) e 3 BEF F = h e f p e b f h b f ( , ) ( | , ) ( , ) ( 3 , 4 ) ( 2 , 3 ) EF b = = h e f p G g e f ( , ) ( | , ) G e ( 4 , 3 ) 4 EFG Time: O ( exp(w+1 )) For each cluster P(X|e) is computed, also P(e) Space: O ( exp(sep)) 95
Example A B C p(a), p(b|a), p(c|a,b) = tree decomposit ion BN X,D,G,P A for a belief network is a BC = T T (V,E) χ ψ triple , , , where is a tree and and are labeling v V χ(v) X functions, associatin g with each verte x two sets, and B C D F p(d|b), p(f|c,d) ψ(v) P satisfying : p P 1. For each function there is exactly one vertex such that BF i ψ(v) χ(v) p scope(p ) and i i B E F χ(v)} X X {v V|X 2. For each varia ble the set forms a p(e|b,f) i i connected subtree (running intersecti on property) EF A B E F G p(g|e,f) E C D Belief network Tree decomposition F 96 G
IJGP - The basic idea • Apply Cluster Tree Elimination to any join-graph • We commit to graphs that are I-maps • Avoid cycles as long as I-mapness is not violated • Result: use minimal arc-labeled join-graphs slides10 828X 2019
Minimal Arc-Labeled Decomposition B C B C ABCDE BCE ABCDE BCE C DE C E DE C E CDEF CDEF a) Fragment of an a) Shrinking labels to make it a arc-labeled join-graph minimal arc-labeled join-graph • Use a DFS algorithm to eliminate cycles relative to each variable slides10 828X 2019
Minimal arc-labeled join-graph
Message propagation B C ABCDE BCE ABCDE h (3,1) (bc) C DE p(a), p(c), p(b|ac), 1 3 BCE C E p(d|abe),p(e|b,c) B C h(3,1)(bc) CDEF FGH h (1,2) C DE C E F F GH 2 CDEF GI FGI GHIJ Minimal arc-labeled: = h de p a p c p b ac p d abe p e bc h bc ( ) ( ) ( ) ( | ) ( | ) ( | ) ( ) sep(1,2)={D,E} ( 1 , 2 ) ( 3 , 1 ) a b c , , elim(1,2)={A,B,C} = h cde p a p c p b ac p d abe p e bc h bc ( ) ( ) ( ) ( | ) ( | ) ( | ) ( ) Non-minimal arc-labeled: ( 1 , 2 ) ( 3 , 1 ) sep(1,2)={C,D,E} a b , elim(1,2)={A,B} slides10 828X 2019
IJGP - Example A C A B C A ABC C A AB BC C D E ABDE BCE BE C DE CE CDEF F H G H F FGH H F FG GH H GI I J FGI GHIJ Belief network Loopy BP graph slides10 828X 2019
Arc-Minimal Join-Graph A A A A A A A A A A A A A C C C C A A A A A A A A A A A A A ABC ABC ABC ABC ABC ABC ABC ABC ABC ABC ABC ABC ABC C C C C C C C C C C C C C A A AB AB AB AB AB AB AB AB AB AB AB AB AB BC BC BC BC BC BC BC BC BC BC BC BC BC C C C C ABDE ABDE ABDE ABDE ABDE ABDE ABDE ABDE ABDE ABDE ABDE ABDE ABDE BCE BCE BCE BCE BCE BCE BCE BCE BCE BCE BCE BCE BCE Arcs labeled with BE BE BE BE BE BE BE BE BE BE BE BE any single variable C C C C C C C C C C C C C DE DE DE DE DE DE DE DE DE DE DE DE DE CE CE CE CE CE CE CE CE CE CE CE CE CE should form a TREE CDEF CDEF CDEF CDEF CDEF CDEF CDEF CDEF CDEF CDEF CDEF CDEF CDEF H H H H H H H H H H H H H F F F F F F F F FGH FGH FGH FGH FGH FGH FGH FGH FGH FGH FGH FGH FGH H H H H H H H H H H H H H F F F F F F F F F F F F F FG FG FG FG FG FG FG FG FG FG FG FG FG GH GH GH GH GH GH GH GH GH GH GH GH GH H H H H H H H H H H GI GI GI GI GI GI GI GI GI GI GI GI GI FGI FGI FGI FGI FGI FGI FGI FGI FGI FGI FGI FGI FGI GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ GHIJ slides10 828X 2019
Collapsing Clusters A A ABC C AB BC ABDE BCE ABCDE ABCDE ABCDE BCE BCE BCE ABCDE BE BC BC BC C DE CE CDE CDE CDE CE CE CE CDE CDEF CDEF CDEF CDEF CDEF H FGH H FGH FGH FGH F F F F F FG GH FG FG FG GH GH GH GI GI GI GI GHI FGI GHIJ FGI FGI FGI GHIJ GHIJ GHIJ FGHI GHIJ slides10 828X 2019
Join-Graphs A C A A ABC C A ABC C BC A AB BC C AB BC ABCDE ABCDE BCE ABDE BCE ABDE BCE BE DE CE CDE C C DE CE DE CE CDEF CDEF CDEF CDEF FGH H H F FGH H FGH H F F F F F GH FG GH H F GH GHI GI GI GI FGI GHIJ FGI GHIJ FGI GHIJ FGHI GHIJ more accuracy less complexity slides10 828X 2019
Bounded decompositions • We want arc-labeled decompositions such that: • the cluster size (internal width) is bounded by i (the accuracy parameter) • Possible approaches to build decompositions: • partition-based algorithms - inspired by the mini-bucket decomposition • grouping-based algorithms slides10 828X 2019
Constructing Join-Graphs A B G: (GFE) P(G|F,E) GFE C D E EF E: (EBF) (EF) EBF P(E|B,F) F G F BF F: ( FCD ) ( BF ) P(F|C,D) FCD BF CD D: (DB) (CD) P(D|B) CDB CB B C: (CAB) (CB) P(C|A,B) CAB BA B: (BA) (AB) (B) P(B|A) BA A A: (A) P(A) A a) schematic mini-bucket(i), i=3 b) arc-labeled join-graph decomposition slides10 828X 2019
IJGP properties • IJGP( i ) applies BP to min arc-labeled join-graph, whose cluster size is bounded by i • On join-trees IJGP finds exact beliefs • IJGP is a Generalized Belief Propagation algorithm (Yedidia, Freeman, Weiss 2001) • Complexity of one iteration: • time: O(deg •( n+N ) •d i+1 ) O( N•d ) • space: slides10 828X 2019
Empirical evaluation ◼ Measures: • Algorithms: ◼ Absolute error • Exact ◼ Relative error • IBP • MC ◼ Kulbach-Leibler (KL) distance • IJGP ◼ Bit Error Rate ◼ Time ◼ Networks (all variables are binary): ◼ Random networks ◼ Grid networks (MxM) ◼ CPCS 54, 360, 422 ◼ Coding networks slides10 828X 2019
Coding Networks – Bit Error Rate σ = .22 σ = .32 Coding, N=400, 1000 instances, 30 it, w*=43, sigma=.22 Coding, N=400, 500 instances, 30 it, w*=43, sigma=.32 1e-1 0.00243 IJGP MC 0.00242 IBP 1e-2 0.00241 IBP IJGP BER BER 1e-3 0.00240 0.00239 1e-4 0.00238 1e-5 0.00237 0 2 4 6 8 10 12 0 2 4 6 8 10 12 i-bound i-bound σ = .51 σ = .65 Coding, N=400, 500 instances, 30 it, w*=43, sigma=.51 Coding, N=400, 500 instances, 30 it, w*=43, sigma=.65 0.0785 0.0780 0.1914 0.0775 0.1912 IBP IBP 0.0770 0.1910 IJGP IJGP BER BER 0.0765 0.1908 0.0760 0.1906 0.0755 0.1904 0.0750 0.1902 0.0745 0.1900 0 2 4 6 8 10 12 0 2 4 6 8 10 12 i-bound i-bound slides10 828X 2019
CPCS 422 – KL Distance CPCS 422, evid=0, w*=23, 1instance CPCS 422, evid=30, w*=23, 1instance 0.1 0.1 IJGP 30 it (at convergence) MC IBP 10 it (at convergence) 0.01 0.01 KL distance KL distance IJGP at convergence MC IBP at convergence 0.001 0.001 0.0001 0.0001 2 4 6 8 10 12 14 16 18 3 4 5 6 7 8 9 10 11 12 13 14 15 16 i-bound i-bound evidence=0 evidence=30 slides10 828X 2019
CPCS 422 – KL vs. Iterations CPCS 422, evid=0, w*=23, 1instance CPCS 422, evid=30, w*=23, 1instance 1 0.1 IJGP (3) IJGP(3) IJGP(10) IJGP(10) IBP IBP 0.1 0.01 KL distance KL distance 0.01 0.001 0.001 0.0001 0.0001 0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35 number of iterations number of iterations evidence=0 evidence=30 slides10 828X 2019
Coding networks - Time Coding, N=400, 500 instances, 30 iterations, w*=43 10 8 IJGP 30 iterations MC IBP 30 iterations 6 Time (seconds) 4 2 0 0 2 4 6 8 10 12 i-bound slides10 828X 2019
More On the Power of Belief Propagation • BP as local minima of KL distance (Read Darwiche) • BP’s power from constraint propagation perspective. 115
Lambda is Grounding for evidence e)
Theorem: Yedidia, Frieman and Weiss 2005
Summary of IJGP so far
Outline • Mini-bucket elimination • Weighted Mini-bucket • Mini-clustering • Iterative Belief propagation • Iterative-join-graph propagation • Re-parameterization, cost-shifting slides10 828X 2019
Cost-Shifting (Reparameterization) + 𝜇(𝐶) − 𝜇(𝐶) A B f(A,B) B C f(B,C) b b 6 + 3 b b 6 – 3 A B C f(A,B,C) b g 0 – 1 b g 0 – 3 b b b 12 g b 0 + 3 g b 0 + 1 b b g 6 g g 6 – 1 g g 6 + 1 b g b 0 b g g 6 = 0 + 6 g b b 6 B λ (B) g b g 0 b 3 Modify the individual functions g g b 6 g -1 g g g 12 - but – keep the sum of functions the same slides10 828X 2019
Tightening the bound • Reparameterization (or, “cost shifting”) A B C F(A,B,C) • Decrease bound without changing overall function 0 0 0 3.0 0 0 1 2.0 A B f 1 (A,B) B C f 2 (B,C) 0 1 0 2.0 0 0 2.0 0 0 1.0 = + 0 1 1 4.0 1 0 3.5 0 1 0.0 1 0 0 4.5 0 1 1.0 1 0 1.0 1 0 1 3.5 1 1 3.0 1 1 3.0 1 1 0 4.0 1 1 1 6.0 A B f 1 (A,B) ¸ (B) B C f 2 (B,C) - ¸ (B) (Adjusting functions 0 0 2.0 0 0 1.0 cancel each other) + 0 0 = 1 0 3.5 0 1 0.0 0 1 1.0 1 0 1.0 +1 -1 (Decomposition bound is exact) 1 1 3.0 1 1 3.0 slides10 828X 2019 127
Dual Decomposition 𝑔 13 (𝑦 1 , 𝑦 3 ) 𝑔 13 (∙) 𝑦 1 𝑦 3 𝑦 1 𝑦 3 𝑦 3 𝑦 1 𝑔 12 (𝑦 1 , 𝑦 2 ) 𝑔 23 (𝑦 2 , 𝑦 3 ) 𝑔 12 (∙) 𝑔 23 (∙) 𝑦 2 𝑦 2 𝑦 2 𝐺 ∗ = min ≥ min 𝑔 𝛽 (𝑦) 𝑦 𝑔 𝛽 (𝑦) 𝑦 𝛽 𝛽 • Bound solution using decomposed optimization • Solve independently: optimistic bound slides10 828X 2019
Dual Decomposition 𝑔 13 (𝑦 1 , 𝑦 3 ) 𝜇 1→13 (𝑦 1 ) 𝜇 3→13 (𝑦 3 ) 𝑔 13 (∙) 𝑦 1 𝑦 3 𝑦 1 𝑦 3 𝜇 3→23 (𝑦 3 ) 𝜇 1→12 (𝑦 1 ) 𝑦 3 𝑦 1 𝑔 12 (𝑦 1 , 𝑦 2 ) 𝑔 23 (𝑦 2 , 𝑦 3 ) 𝑔 12 (∙) 𝑔 23 (∙) Reparameterization: 𝑦 2 𝑦 2 𝑦 2 ∀𝑘 ∶ 𝜇 𝑘→𝛽 𝑦 𝑘 = 0 𝜇 2→13 (𝑦 2 ) 𝜇 2→23 (𝑦 2 ) 𝛽∋𝑘 𝐺 ∗ = min max + 𝜇 𝑗→𝛽 𝑦 𝑗 ≥ min 𝑔 𝛽 (𝑦) 𝑦 𝑔 𝛽 (𝑦) 𝜇 𝑗→𝛽 𝑦 𝑗∈𝛽 𝛽 𝛽 • Bound solution using decomposed optimization • Solve independently: optimistic bound • Tighten the bound by reparameterization ‒ Enforce lost equality constraints via Lagrange multipliers slides10 828X 2019
Dual Decomposition 𝑔 13 (𝑦 1 , 𝑦 3 ) 𝜇 1→13 (𝑦 1 ) 𝜇 3→13 (𝑦 3 ) 𝑔 13 (∙) 𝑦 1 𝑦 3 𝑦 1 𝑦 3 𝜇 3→23 (𝑦 3 ) 𝜇 1→12 (𝑦 1 ) 𝑦 3 𝑦 1 𝑔 12 (𝑦 1 , 𝑦 2 ) 𝑔 23 (𝑦 2 , 𝑦 3 ) 𝑔 12 (∙) 𝑔 23 (∙) Reparameterization: 𝑦 2 𝑦 2 𝑦 2 ∀𝑘 ∶ 𝜇 𝑘→𝛽 𝑦 𝑘 = 0 𝜇 2→13 (𝑦 2 ) 𝜇 2→23 (𝑦 2 ) 𝛽∋𝑘 𝐺 ∗ = min max + 𝜇 𝑗→𝛽 𝑦 𝑗 ≥ min 𝑔 𝛽 (𝑦) 𝑦 𝑔 𝛽 (𝑦) 𝜇 𝑗→𝛽 𝑦 𝑗∈𝛽 𝛽 𝛽 Many names for the same class of bounds: ‒ Dual decomposition [Komodakis et al. 2007] ‒ TRW, MPLP [Wainwright et al. 2005; Globerson & Jaakkola, 2007] ‒ Soft arc consistency [Cooper & Schiex, 2004] ‒ Max-sum diffusion [Warner 2007] slides10 828X 2019
Dual Decomposition 𝑔 13 (𝑦 1 , 𝑦 3 ) 𝜇 1→13 (𝑦 1 ) 𝜇 3→13 (𝑦 3 ) 𝑔 13 (∙) 𝑦 1 𝑦 3 𝑦 1 𝑦 3 𝜇 3→23 (𝑦 3 ) 𝜇 1→12 (𝑦 1 ) 𝑦 3 𝑦 1 𝑔 12 (𝑦 1 , 𝑦 2 ) 𝑔 23 (𝑦 2 , 𝑦 3 ) 𝑔 12 (∙) 𝑔 23 (∙) Reparameterization: 𝑦 2 𝑦 2 𝑦 2 ∀𝑘 ∶ 𝜇 𝑘→𝛽 𝑦 𝑘 = 0 𝜇 2→13 (𝑦 2 ) 𝜇 2→23 (𝑦 2 ) 𝛽∋𝑘 𝐺 ∗ = min max + 𝜇 𝑗→𝛽 𝑦 𝑗 ≥ min 𝑔 𝛽 (𝑦) 𝑦 𝑔 𝛽 (𝑦) 𝜇 𝑗→𝛽 𝑦 𝑗∈𝛽 𝛽 𝛽 Many ways to optimize the bound: ‒ Sub-gradient descent [Komodakis et al. 2007; Jojic et al. 2010] ‒ Coordinate descent [Warner 2007; Globerson & Jaakkola 2007; Sontag et al. 2009; Ihler et al. 2012] ‒ Proximal optimization [Ravikumar et al, 2010] ‒ ADMM [Meshi & Globerson 2011; Martins et al. 2011; Forouzan & Ihler 2013] slides10 828X 2019
Recommend
More recommend
