Bayesian estimation of the latent dimension and communities in - PowerPoint PPT Presentation

Bayesian community detection Results Conclusion References JSM19 – Novel Approaches for Analyzing Dynamic Networks Bayesian estimation of the latent dimension and communities in stochastic blockmodels Francesco Sanna Passino , Nick Heard Department of Mathematics, Imperial College London francesco.sanna-passino16@imperial.ac.uk July 30, 2019 1/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Stochastic blockmodels as random dot product graphs Consider an undirected graph with symmetric adjacency matrix A ∈ { 0 , 1 } n × n . In random dot product graphs , the probability of a link between two nodes is expressed as the inner product between two latent positions x i , x j ∈ F , 0 ≤ x ⊤ y ≤ 1 ∀ x , y ∈ F : P ( A ij = 1) = x ⊤ i x j . The stochastic blockmodel is the classical model for community detection in graphs. Given a matrix B ∈ [0 , 1] K × K of within-community probabilities, the probability of a link depends on the community allocations z i and z j ∈ { 1 , . . . , K } of the two nodes: P ( A ij = 1) = B z i z j . The stochastic blockmodel can be interpreted as a special case of a random dot product graph. If B kh = µ ⊤ k µ h with µ k , µ h ∈ F , and all the nodes in community k are assigned the latent position µ k , then: P ( A ij = 1) = µ ⊤ z i µ z j , i < j, A ij = A ji . 2/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Network embeddings Consider an undirected graph with symmetric adjacency matrix A ∈ { 0 , 1 } n × n , and modified Laplacian L = D − 1 / 2 AD − 1 / 2 , D = diag( � n i =1 A ij ) . The adjacency embedding of A in R d is: x n ] ⊤ = ˆ Λ 1 / 2 ∈ R n × d , ˆ Γ ˆ X = [ˆ x 1 , . . . , ˆ where ˆ Λ is a d × d diagonal matrix containing the top d largest eigenvalues of A , and ˆ Γ is a n × d matrix containing the corresponding orthonormal eigenvectors. The Laplacian embedding of A in R d is: x n ] ⊤ = ˜ Λ 1 / 2 ∈ R n × d , ˜ Γ ˜ X = [˜ x 1 , . . . , ˜ where ˜ Λ is a d × d diagonal matrix containing the top d largest eigenvalues of L , and ˜ Γ is a n × d matrix containing the corresponding orthonormal eigenvectors. 3/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Spectral estimation of the stochastic blockmodel Based on asymptotic properties, Rubin-Delanchy et al., 2017, propose the following algorithm for consistent estimation of the latent positions in stochastic blockmodels: Algorithm 1: Spectral estimation of the stochastic blockmodel (spectral clustering) Input: adjacency matrix A (or the Laplacian matrix L ), dimension d , and number of communities K ≥ d . x n ] ⊤ or ˜ x n ] ⊤ into R d , 1 compute spectral embedding ˆ X = [ˆ x 1 , . . . , ˆ X = [˜ x 1 , . . . , ˜ 2 fit a Gaussian mixture model with K components, Result: return cluster centres µ 1 , . . . , µ K ∈ R d and node memberships z 1 , . . . , z n . In practice: d and K are estimated sequentially . Issues: Sequential approach is sub-optimal : the estimate of K depends on choice of d . Theoretical results only hold for d fixed and known . Distributional assumptions when d is misspecified are not available . This talk discusses a novel framework for joint estimation of d and K . 4/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References A Bayesian model for network embeddings Choose integer m ≤ n and obtain embedding X ∈ R n × m → m arbitrarily large. Bayesian model for simultaneous estimation of d and K → allow for d = rank ( B ) ≤ K . �� µ z i � � Σ z i �� 0 d x i | d, z i , µ z i , Σ z i , σ 2 ∼ N m , , i = 1 , . . . , n, z i σ 2 0 m − d 0 z i I m − d ( µ k , Σ k ) | d iid ∼ NIW d ( 0 , κ 0 , ν 0 + d − 1 , ∆ d ) , k = 1 , . . . , K, iid σ 2 ∼ Inv - χ 2 ( λ 0 , σ 2 0 ) , j = d + 1 , . . . , m, kj d | z d ∼ Uniform { 1 , . . . , K ∅ } , z i | θ iid ∼ Multinoulli( θ ) , i = 1 , . . . , n, θ ∈ S K − 1 , � α K , . . . , α � θ | K d ∼ Dirichlet , K K d ∼ Geometric( ω ) . where K ∅ is the number of non-empty communities. Alternative: d d ∼ Geometric( δ ) . Yang et al., 2019, independently proposed a similar frequentist model. 5/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Empirical model validation 0 . 5 0 . 4 0 . 2 0 0 − 0 . 2 − 0 . 5 − 0 . 4 − 0 . 4 − 0 . 2 0 0 . 2 0 . 4 0 . 6 − 1 − 0 . 9 − 0 . 8 − 0 . 7 − 0 . 6 − 0 . 5 Figure 2. Scatterplot of the simulated X 3 and X 4 Figure 1. Scatterplot of the simulated X 1 and X 2 – i.e. X : d Simulated GRDPG-SBM with n = 2500 , d = 2 , K = 5 . Nodes allocated to communities with probability θ k = P ( z i = k ) = 1 /K . 6/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Empirical model validation 0 . 5 15 Overall mean ρ ( k ) ij for X : d Max/min within-cluster mean Histogram of ρ ( k ) ij for X d : Within-cluster mean 10 0 5 − 0 . 5 0 − 1 − 0 . 2 − 0 . 1 0 0 . 1 0 . 2 2 4 6 8 10 12 14 Correlation coefficient ρ ( k ) Dimension ij Figure 3. Within-cluster and overall means of X :15 Figure 4. Within-cluster correlation coefficients of X :30 Means are approximately 0 for columns with index > d . Reasonable to assume correlation ρ ( k ) ij = 0 for i, j > d . 7/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Curse of dimensionality · 10 − 2 · 10 − 2 Within cluster variance 6 6 Total variance 4 Variance Variance 4 2 2 0 0 0 5 10 15 20 25 0 100 200 300 400 500 Dimension Dimension Figure 5. Within-block variance and total variance for the adjacency embedding obtained from a simulated SBM with d = 2 , K = 5 , n = 500 , and well separated means µ 1 = [0 . 7 , 0 . 4] , µ 2 = [0 . 1 , 0 . 1] , µ 3 = [0 . 4 , 0 . 8] , µ 4 = [ − 0 . 1 , 0 . 5] and µ 5 = [0 . 3 , 0 . 5] , and θ = (0 . 2 , 0 . 2 , 0 . 2 , 0 . 2 , 0 . 2) . For some k and k ′ : σ 2 kj ≈ σ 2 k ′ j for j ≫ d and k � = k ′ . 8/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Second order clustering Bayesian model parsimony: K underestimated for d ≪ m . Possible solution: second order clustering v = ( v 1 , . . . , v K ) with v k ∈ { 1 , . . . , H } . If v k = v k ′ , then σ 2 kj = σ 2 k ′ j for j > d : �� µ z i � � Σ z i �� 0 d x i | d, z i , v z i , µ z i , Σ z i , σ 2 ∼ N m , , i = 1 , . . . , n, σ 2 v zi 0 v zi I m − d 0 m − d v k | K, H d ∼ Multinoulli( φ ) , k = 1 , . . . , K, � β � H , . . . , β φ | H d ∼ Dirichlet , H H | K d ∼ Uniform { 1 , . . . , K } . The parameter v k defines clusters of clusters . Empirical results show that the model is able to handle d ≪ m . 9/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References N 3 ( µ 1 , Σ 1 ) N 8 ( 0 8 , σ 2 1 I 5 ) N 3 ( µ 2 , Σ 2 ) N 3 ( µ 3 , Σ 3 ) N 8 ( 0 8 , σ 2 2 I 5 ) N 3 ( µ 4 , Σ 4 ) N 3 ( µ 5 , Σ 5 ) N 8 ( 0 8 , σ 2 3 I 5 ) 3 = latent dimension 10/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels

Bayesian community detection Results Conclusion References Extension to directed and bipartite graphs Consider a directed graph with adjacency matrix A ∈ { 0 , 1 } n × n . The d -dimensional adjacency embedding of A in R 2 d , is defined as: � ˆ � ˆ D 1 / 2 ⊕ ˆ D 1 / 2 = X = ˆ ˆ U ˆ V ˆ U ˆ V ˆ ˆ ˆ D 1 / 2 D 1 / 2 � � = X s X r . V ⊤ + ˆ where A = ˆ U ˆ D ˆ U ⊥ ˆ D ⊥ ˆ ⊥ is the SVD decomposition of A , where ˆ V ⊤ U ∈ R n × d , ˆ diagonal, and ˆ D ∈ R d × d V ∈ R n × d . + Essentially, only three distributions change:       µ z i Σ z i 0 0 0 σ 2 0 m − d 0 z i I m − d 0 0 d       x i | d, K, z i ∼ N 2 m  ,  ,       µ ′ Σ ′ 0 0 0   z i  z i  σ 2 ′ 0 m − d 0 0 0 z i I m − d ( µ k , Σ k ) | d, K iid ∼ NIW d ( 0 , κ 0 , ν 0 + d − 1 , ∆ d ) , k = 1 , . . . , K, kj | d, K iid σ 2 ∼ Inv - χ 2 ( λ 0 , σ 2 0 ) , j = d + 1 , . . . , m. Co-clustering : different clusters for sources and receivers → bipartite graphs. 11/16 Francesco Sanna Passino Imperial College London Bayesian estimation of the latent dimension and communities in stochastic blockmodels