Bayesian Methods for Graph Clustering P. Latouche, E. Birmel e - PowerPoint PPT Presentation

Bayesian Methods for Graph Clustering P. Latouche, E. Birmel´ e Laboratoire ”Statistique et G´ enome” (UMR CNRS 8071, INRA 1152) Journ´ ees MAS, August 2008 P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 1 / 22

Outline Introduction 1 Real networks Random graph models The MixNet model Maximum likelihood estimation Bayesian View of MixNet 2 Bayesian probabilistic model Variational inference Model selection Applications 3 Affiliation models Metabolic network of E. coli P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 2 / 22

Outline Introduction 1 Real networks Random graph models The MixNet model Maximum likelihood estimation Bayesian View of MixNet 2 Bayesian probabilistic model Variational inference Model selection Applications 3 Affiliation models Metabolic network of E. coli P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 3 / 22

Real networks Many scientific fields : World Wide Web, Biology, sociology, physics. Nature of data under study : interactions between n objects, O ( n 2 ) possible interactions. Network topology : describes the way nodes interact, structure/function Sample of 250 blogs (nodes) with their links (edges) relationship. of the French political Blogosphere. P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 4 / 22

Estimation of random graph models Random graph models Random graph approaches (Govaert 1977, Frank and Harary 1982, Handcock 2006, Newman and Leicht 2007, Hofman and Wiggins 2008). The Block-Clustering model (Snijders and Nowicki 1997). Erd¨ os-R´ enyi Mixture Model for Network (MixNet; Daudin et al 2008). Estimation of the model parameters Bayesian strategies cannot handle large networks. Maximum likelihood strategies. P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 5 / 22

The MixNet probabilistic model Origin model developed by J. Daudin et al. (2008), ER model generalization, application fields: biology, internet, social network... Modelling connection heterogeneity hyp.: there exists a hidden structure with Q classes, Z = ( Z i ) i , Z iq = I { i ∈ q } are indep. hidden variables, α = { α q } , the prior proportions of groups, ( Z i ) ∼ M (1 , α ). Distribution of X k Conditional distribution: X ij |{ Z iq Z j ℓ = 1 } ∼ B ( π ql ) X jk=0 where B () is the Bernoulli distribution , j X ij=1 X ij | Z are independant. i Marginal distribution: X ij ∼ � q ℓ α q α ℓ B ( π ql ), P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 6 / 22

Maximum likelihood estimation Likelihood(s) of the model : → Observed data : p ( X | α , π ) = � Z p ( X , Z | α , π ). → Complete data : p ( X , Z | α , π ). → EM-like strategies require the knowledge of p ( Z | X , α , π ). Problem In our case, p ( Z | X , α , π ) is not tractable (no conditional independence). P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 7 / 22

Variational method Daudin et. al, 2008 Decomposition and variational EM � � � � ln p ( X | α , π ) = L q ( . ); α , π + KL q ( . ) || p ( . | X , α , π ) , where q ( Z ) ln { p ( X , Z | α , π ) � � � L q ( . ); α , π = } , q ( Z ) Z and q ( Z ) ln { p ( Z | X , α , π ) � � � KL q ( . ) || p ( . | X , α , π ) = − } . q ( Z ) Z Approximation N N � � q ( Z ) = q ( Z i ) = M ( Z i ; 1 , τ i ) . i =1 i =1 P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 8 / 22

Estimation of the number of classes Criteria Since p ( X | α , π ) is not tractable, many criteria cannot be used: 1 Akaike Information Criterion: AIC = ln p ( X | α ML , π ML ) − M . 2 Bayesian Information Criterion: BIC = ln p ( X | α MAP , π MAP ) − 1 2 M ln n . Integrated Classification Likelihood (ICL) 1 Following the work of Biernacki et al. (2000), Mariadassou and Robin (2007) used a criterion based on an asymptotic approximation of the Integrated Classification Likelihood (ICL). � � 2 ICL = max α , π ln p ( X , ˜ Q ( Q +1) Z | α , π ) − 1 � � ln n ( n +1) − ( Q − 1) ln( n ) 2 2 P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 9 / 22

Outline Introduction 1 Real networks Random graph models The MixNet model Maximum likelihood estimation Bayesian View of MixNet 2 Bayesian probabilistic model Variational inference Model selection Applications 3 Affiliation models Metabolic network of E. coli P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 10 / 22

Conjugate prior distributions Mixing coefficients : α ∼ Dirichlet ( α ; n 0 ) n 0 = ( n 0 1 , . . . , n 0 → Q ). n 0 → q is the prior number of vertices in class q . Connectivity matrix : π ∼ � Q q , l Beta( π ql ; η 0 ql , ζ 0 ql ) η 0 → ql is the prior number of edges connecting vertices of class q to vertices of class l . ζ 0 → ql is the prior number of non -edges connecting vertices of class q to vertices of class l . P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 11 / 22

Variational Bayes Decomposition � � � � ln p ( X ) = L q ( . ) + KL q ( . ) || p ( . | X , where q ( Z , α , π ) ln { p ( X , Z , α , π ) � � � � � L q ( . ) = q ( Z , α , π ) } d α d π , Z and � � q ( Z , α , π ) ln { p ( Z , α , π | X ) � � � KL q ( . ) || p ( . | X ) = − q ( Z , α , π ) } d α d π . Z P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 12 / 22

Variational Bayes Factorization N � q ( Z , α , π ) = q ( α ) q ( π ) q ( Z ) = q ( α ) q ( π ) q ( Z i ) . i =1 Optimization 1 ln ˜ q ( Z i ) = E Z \ i , α , π [ln p ( X , Z , α , π )] + cste . 2 ln ˜ q ( α ) = E Z , π [ln p ( X , Z , α , π )] + cste . 3 ln ˜ q ( π ) = E Z , α [ln p ( X , Z , α , π )] + cste . P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 13 / 22

Optimization Variational Bayes E-step q ( Z i ) = M ( Z i ; 1 , τ i = { τ i 1 , . . . , τ iQ } ) . Variational Bayes M-step (1) q ( α ) = Dir( α ; n ) , q + � N where n q = n 0 i =1 τ iq . Variational Bayes M-step (2) Q � q ( π ) = Beta( π ql | η ql , ζ ql ) , q , l where η ql = η 0 ql + � N i � = j X ij τ iq τ jl and ζ ql = ζ 0 ql + � N i � = j (1 − X ij ) τ iq τ jl . P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 14 / 22

Model selection The model evidence depends on Q . Bayes’ rule leads to p ( Q | X ) ∝ p ( X | Q ) p ( Q ). If p ( Q ) is broad, maximizing p ( Q | X ) is equivalent to maximizing p ( X | Q ). Since p ( X | Q ) is intractable we propose to use the lower bound � � L q ( . ) and to add a term ln Q ! to take the multimodality into account . First non-asymptotic criterion based on an approximation of the model evidence. P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 15 / 22

Outline Introduction 1 Real networks Random graph models The MixNet model Maximum likelihood estimation Bayesian View of MixNet 2 Bayesian probabilistic model Variational inference Model selection Applications 3 Affiliation models Metabolic network of E. coli P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 16 / 22

Experiments on affiliation models Probability of intra-connection : λ . Probability of inter-connection : ǫ . Number of vertices : n = 50 . For each graph model ( λ + ǫ = 1 ) and for each number of classes Q True ∈ { 2 , 3 , 4 , 5 } , we generated 100 graphs . 5 initializations using spectral clustering techniques . Select the best number of estimated classes according to each criterion . P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 17 / 22

Affiliation model (1) 1 2 3 4 5 6 2 0 100 0 0 0 0 a ) Q True / Q ICL 3 0 0 100 0 0 0 4 0 0 1 98 1 0 5 0 0 10 61 29 0 1 2 3 4 5 6 2 0 100 0 0 0 0 b ) Q True / Q VB 3 0 0 100 0 0 0 4 0 0 0 98 2 0 5 0 0 1 29 65 5 Table: λ = 0 . 85 and ǫ = 0 . 15. P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 18 / 22

Affiliation model (2) 1 2 3 4 5 6 2 0 100 0 0 0 0 a ) Q True / Q ICL 3 0 0 100 0 0 0 4 0 0 14 86 0 0 5 0 17 36 44 3 0 1 2 3 4 5 6 2 0 100 0 0 0 0 b ) Q True / Q VB 3 0 0 100 0 0 0 4 0 0 5 94 1 0 5 0 4 18 43 29 6 Table: λ = 0 . 8 and ǫ = 0 . 2. P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 19 / 22

Metabolic network of E. coli 593 vertices . 1782 edges . 60 initializations . Compute the lower bound . P. Latouche (Stat. & G´ enome) Bayesian Methods for Graph Clustering Journ´ ees MAS, August 2008 20 / 22