Variational methods for overlapping and non-overlapping stochastic - PowerPoint PPT Presentation

Variational methods for overlapping and non-overlapping stochastic block models Pierre Latouche Universit´ e Paris 1 Panth´ eon-Sorbonne Laboratoire SAMM MSTGA 2012 Pierre Latouche 1

Contents Introduction Real networks Graph clustering Stochastic block models Model selection The overlapping stochastic block model Model selection Bayesian framework Inference The regulation term β Model selection Experiments Simulated data The French blogosphere network Pierre Latouche 2

Real networks ◮ Many scientific fields : ◮ World Wide Web ◮ Biology, sociology, physics ◮ Nature of data under study : ◮ Interactions between N objects ◮ O ( N 2 ) possible interactions ◮ Network topology : ◮ Describes the way nodes interact, structure/function Sample of 250 blogs (nodes) with their links relationship (edges) of the French political Blogosphere. Pierre Latouche 3

In Biology The metabolic network of bacteria Escherichia coli (Lacroix et al., 2006). Pierre Latouche 4

In Biology Subset of the yeast transcriptional regulatory network (Milo et al., 2002). Pierre Latouche 5

Real networks ◮ Properties : ◮ Sparsity : m = O( N ) ◮ Existence of a giant component ◮ Heterogeneity ◮ Preferential attachment ◮ Small world → Topological structure (groups of vertices) ֒ Pierre Latouche 6

Real networks ◮ Properties : ◮ Sparsity : m = O( N ) ◮ Existence of a giant component ◮ Heterogeneity ◮ Preferential attachment ◮ Small world → Topological structure (groups of vertices) ֒ Pierre Latouche 6

Graph clustering ◮ Existing methods look for : ◮ Community structure ◮ Disassortative mixing ◮ Heterogeneous structure Pierre Latouche 7

Graph clustering ◮ Existing methods look for : ◮ Community structure ◮ Disassortative mixing ◮ Heterogeneous structure Pierre Latouche 7

Graph clustering ◮ Existing methods look for : ◮ Community structure ◮ Disassortative mixing ◮ Heterogeneous structure Pierre Latouche 7

Graph clustering ◮ Existing methods look for : ◮ Community structure ◮ Disassortative mixing ◮ Heterogeneous structure Pierre Latouche 7

Stochastic Block Model (SBM) ◮ Nowicki and Snijders (2001) ◮ Earlier work : Govaert et al. (1977) ◮ Z i independent hidden variables : � � ◮ Z i ∼ M 1 , α = ( α 1 , α 2 , . . . , α K ) ◮ Z ik = 1 : vertex i belongs to class k ◮ X | Z edges drawn independently : X ij |{ Z ik Z jl = 1 } ∼ B ( π kl ) ◮ A mixture model for graphs : K K � � X ij ∼ α k α l B ( π kl ) k =1 l =1 Pierre Latouche 8

π •• 6 6 π •• 5 5 7 7 3 π •• 4 4 8 8 1 2 π •• 10 π •• 9 Pierre Latouche 9

Maximum likelihood estimation ◮ Log-likelihoods of the model : ◮ Observed-data : log p ( X | α , Π ) = log { � Z p ( X , Z | α , Π ) } → K N terms ֒ ◮ Expectation Maximization (EM) algorithm requires the knowledge of p ( Z | X , α , Π ) Problem p ( Z | X , α , Π ) is not tractable (no conditional independence) Variational EM Daudin et al. (2008) Pierre Latouche 10

Maximum likelihood estimation ◮ Log-likelihoods of the model : ◮ Observed-data : log p ( X | α , Π ) = log { � Z p ( X , Z | α , Π ) } → K N terms ֒ ◮ Expectation Maximization (EM) algorithm requires the knowledge of p ( Z | X , α , Π ) Problem p ( Z | X , α , Π ) is not tractable (no conditional independence) Variational EM Daudin et al. (2008) Pierre Latouche 10

Maximum likelihood estimation ◮ Log-likelihoods of the model : ◮ Observed-data : log p ( X | α , Π ) = log { � Z p ( X , Z | α , Π ) } → K N terms ֒ ◮ Expectation Maximization (EM) algorithm requires the knowledge of p ( Z | X , α , Π ) Problem p ( Z | X , α , Π ) is not tractable (no conditional independence) Variational EM Daudin et al. (2008) Pierre Latouche 10

Model selection Criteria Since log p ( X | α , Π ) is not tractable, we cannot rely on: α , ˆ ◮ AIC = log p ( X | ˆ Π ) − C α , ˆ 2 log N ( N − 1) Π ) − C ◮ BIC = log p ( X | ˆ 2 ICL Biernacki et al. (2000) ֒ → Daudin et al. (2008) Variational Bayes EM ֒ → ILvb Latouche et al. (2012) Pierre Latouche 11

Model selection Criteria Since log p ( X | α , Π ) is not tractable, we cannot rely on: α , ˆ ◮ AIC = log p ( X | ˆ Π ) − C α , ˆ 2 log N ( N − 1) Π ) − C ◮ BIC = log p ( X | ˆ 2 ICL Biernacki et al. (2000) ֒ → Daudin et al. (2008) Variational Bayes EM ֒ → ILvb Latouche et al. (2012) Pierre Latouche 11

Bayesian framework ◮ Conjugate prior distributions : � � α | n 0 = { n 0 ◮ p 1 , . . . , n 0 = Dir( α ; n 0 ) K } � � kl ) , ζ 0 = ( ζ 0 Π | η 0 = ( η 0 = � ◮ p k ≤ l Beta( π kl ; η 0 kl , ζ 0 kl ) kl ) ◮ Non informative Jeffreys prior : ◮ n 0 k = 1 / 2 ◮ η 0 kl = ζ 0 kl = 1 / 2 Pierre Latouche 12

Variational Bayes EM Latouche et al. (2009) ◮ p ( Z , α , Π | X ) not tractable Decomposition log p ( X ) = L ( q ) + KL ( q ( · ) || p ( ·| X )) where � p ( X , Z , α , Π ) � � � � L ( q ) = q ( Z , α , Π ) log d α d Π q ( Z , α , Π ) Z Factorization N � q ( Z , α , Π ) = q ( α ) q ( Π ) q ( Z ) = q ( α ) q ( Π ) q ( Z i ) i =1 Pierre Latouche 13

Variational Bayes EM Latouche et al. (2009) E-step ◮ q ( Z i ) = M ( Z i ; 1 , τ i = { τ i 1 , . . . , τ iK } ) M-step ◮ q ( α ) = Dir( α ; n ) ◮ q ( Π ) = � K k ≤ l Beta( π kl ; η kl , ζ kl ) Pierre Latouche 14

A new model selection criterion : ILvb Latouche et al. (2012) ◮ log p ( X | K ) = L ( q ) + KL( ... ) ◮ After convergence, use L ( q ) as an approximation of log p ( X | K ) ILvb � � Γ( � K k ) � K k =1 n 0 k =1 Γ( n k ) IL vb = log Γ( � K k =1 n k ) � K k =1 Γ( n 0 k ) K � Γ( η 0 � N K kl + ζ 0 kl )Γ( η kl )Γ( ζ kl ) � � � − + log τ ik log τ ik Γ( η kl + ζ kl )Γ( η 0 kl )Γ( ζ 0 kl ) k ≤ l i =1 k =1 Pierre Latouche 15

Contents Introduction Real networks Graph clustering Stochastic block models Model selection The overlapping stochastic block model Model selection Bayesian framework Inference The regulation term β Model selection Experiments Simulated data The French blogosphere network Pierre Latouche 16

Overlaps in networks Palla et al. (2006) Problem The stochastic block model (SBM) and most existing methods assume that each vertex belongs to a single class Pierre Latouche 17

Stochastic Block Model (SBM) ◮ Nowicki and Snijders (2001) ◮ Z i independent hidden variables : � � Z i ∼ M 1 , α = ( α 1 , α 2 , . . . , α K ) Pierre Latouche 18

Overlapping Stochastic Block model (OSBM) ◮ Latouche et al. (2011) ◮ Z ik independent hidden variables : K K � � α Z ik (1 − α k ) 1 − Z ik Z i ∼ B ( Z ik ; α k ) = k k =1 k =1 Pierre Latouche 18

Overlapping Stochastic Block model (OSBM) ◮ Latouche et al. (2011) ◮ X | Z edges drawn independently : � � X ij | Z i , Z j ∼ B X ij ; Π Z i , Z j ) � � ◮ Π Z i , Z j = g a Z i , Z j + V ⊺ Z j + W ∗ ◮ a Z i , Z j = Z ⊺ + Z ⊺ i W Z j i U ���� � �� � � �� � � �� � bias i ↔ j i → ? ? → j ◮ g ( t ) = 1 / (1 + exp( − t )) is the logistic function Pierre Latouche 18

OSBM ◮ ˜ Z i = ( Z i , 1) ⊺ � W � U ˜ W = ◮ W ∗ V ⊺ ◮ a Z i , Z j = ˜ ⊺ i ˜ W ˜ Z Z j � � α , ˜ ◮ Parameter set : W Pierre Latouche 19

Contents Introduction Real networks Graph clustering Stochastic block models Model selection The overlapping stochastic block model Model selection Bayesian framework Inference The regulation term β Model selection Experiments Simulated data The French blogosphere network Pierre Latouche 20

Bayesian framework ◮ Conjugate prior distributions : ◮ p ( α ) = � K k =1 Beta( α k ; η 0 k , ζ 0 k ) vec ) = N ( ˜ vec ; ˜ vec ◮ p ( ˜ 0 , S 0 ) W W W ◮ The vec operator : if � A 11 � A 12 A = , A 21 A 22 then   A 11 A 21   A vec =   A 12   A 22 Pierre Latouche 21

Bayesian framework ◮ x ⊺ A y = ( y ⊗ x ) ⊺ A vec vec ◮ In practice : set ˜ = 0 and S 0 = I W 0 β Problem p ( Z , α , ˜ W | X ) not tractable Pierre Latouche 22

q Transformation Decomposition log p ( X ) = L ( r ) + KL( r || p ) where � � p ( X | Z , ˜ W ) p ( Z | α ) p ( α ) p ( ˜ W ) � � r ( Z , α , ˜ d α d ˜ L ( r ) = W ) log W r ( Z , α , ˜ W ) Z Lower bound log p ( X ) ≥ L ( r ) Problem L ( r ) has a too complex form ֒ → no variational Bayes EM algorithm ?? Pierre Latouche 23

Local bound ◮ Use the bound of Jaakkola and Jordan (2000) for Bayesian logistic regression log p ( X | Z , ˜ W ) ≥ log h ( Z , ˜ W , ξ ) , ∀ ξ ∈ R N × N where N � ( X ij − 1 2) a Z i , Z j − ξ ij � log h ( Z , ˜ W , ξ ) = 2 + log g ( ξ ij ) i � = j � − λ ( ξ ij )( a 2 Z i , Z j − ξ 2 ij ) and λ ( ξ ) = 1 4 ξ tanh( ξ 2) = 1 g ( ξ ) − 1 � � 2 ξ 2 Pierre Latouche 24

ξ Transformation Lower Bound �� � � p ( X | Z , ˜ W ) p ( Z | α ) p ( α ) p ( ˜ W ) d α d ˜ log p ( X ) = log W Z ≥ L ( ξ ) where �� � � h ( Z , ˜ W , ξ ) p ( Z | α ) p ( α ) p ( ˜ W ) d α d ˜ L ( ξ ) = log W Z Pierre Latouche 25