Sparse random graphs with exchangeable point processes Fran cois - PowerPoint PPT Presentation

Sparse random graphs with exchangeable point processes Fran¸ cois Caron Department of Statistics, Oxford Statistics Seminar, Bocconi University March 26, 2015 Joint work with Emily Fox (U. Washington) F. Caron 1 / 57

Introduction Exchangeable matrices and their limitations Statistical network models using exchangeable random measures Exchangeability and sparsity properties Special case: Generalized gamma process Posterior characterization & Inference Experimental results F. Caron 2 / 57

Outline Introduction Exchangeable matrices and their limitations Statistical network models using exchangeable random measures Exchangeability and sparsity properties Special case: Generalized gamma process Posterior characterization & Inference Experimental results F. Caron 3 / 57

Introduction 4 2 3 2 3 1 2 3 1 1 ◮ Multi-edges directed graphs ◮ Emails ◮ Citations ◮ WWW ◮ Simple graphs ◮ Social network ◮ Protein-protein interaction F. Caron 4 / 57

Introduction Readers/Customers A 1 A 2 B 1 B 2 B 3 B 4 ◮ Bipartite graphs ◮ Scientists authoring papers ◮ Readers reading books ◮ Internet users posting messages on forums ◮ Customers buying items F. Caron 5 / 57

Introduction ◮ Build a statistical model of the network to ◮ Find interpretable structure in the network ◮ Predict missing edges ◮ Predict connections of new nodes F. Caron 6 / 57

Introduction ◮ Properties of real world networks ◮ Sparsity Dense graph: n e = Θ( n 2 ) Sparse graph: n e = o ( n 2 ) with n e the number of edges and n the number of nodes ◮ Power-law degree distributions [Newman, 2009, Clauset et al., 2009] F. Caron 7 / 57

Book-crossing community network 5 000 readers, 36 000 books, 50 000 edges F. Caron 8 / 57

Book-crossing community network Degree distributions on log-log scale 0 0 10 10 −1 −1 10 10 −2 −2 10 10 Distribution Distribution −3 −3 10 10 −4 −4 10 10 −5 −5 10 10 −6 −6 10 10 −7 −7 10 10 0 1 2 3 4 0 1 2 10 10 10 10 10 10 10 10 Degree Degree (a) Readers (b) Books F. Caron 9 / 57

Outline Introduction Exchangeable matrices and their limitations Statistical network models using exchangeable random measures Exchangeability and sparsity properties Special case: Generalized gamma process Posterior characterization & Inference Experimental results F. Caron 10 / 57

Introduction ◮ Statistical network modeling ◮ Probabilistic symmetry: exchangeability ◮ Ordering of the nodes is irrelevant 2 3 1 F. Caron 11 / 57

Introduction ◮ Statistical network modeling ◮ Probabilistic symmetry: exchangeability ◮ Ordering of the nodes is irrelevant 3 1 2 F. Caron 12 / 57

Introduction ◮ Graphs usually represented by a discrete structure ◮ Adjacency matrix X ij ∈ { 0 , 1 } , ( i, j ) ∈ N 2 ◮ Joint exchangeability d ( X ij ) = ( X π ( i ) π ( j ) ) for any permutation π of N          π         � �� � π F. Caron 13 / 57

Introduction ◮ Aldous-Hoover representation theorem ( X ij ) = ( F ( U i , U j , U { ij } )) where U i , U { ij } are uniform random variables and F is a random function from [0 , 1] 3 to { 0 , 1 } ◮ Several network models fit in this framework (e.g. stochastic blockmodel, infinite relational model, etc.) [Hoover, 1979, Aldous, 1981, Lloyd et al., 2012] F. Caron 14 / 57

Introduction ◮ Corollary of A-H theorem Exchangeable random graphs are either empty or dense ◮ To quote the survey paper of Orbanz and Roy “the theory [...] clarifies the limitations of exchangeable models. It shows, for example, that most Bayesian models of network data are inherently misspecified” ◮ Give up exchangeability for sparsity? e.g. preferential attachment model [Barab´ asi and Albert, 1999, Orbanz and Roy, 2015] F. Caron 15 / 57

Outline Introduction Exchangeable matrices and their limitations Statistical network models using exchangeable random measures Exchangeability and sparsity properties Special case: Generalized gamma process Posterior characterization & Inference Experimental results F. Caron 16 / 57

Point process representation ◮ Representation of a graph as a (marked) point process over R 2 + ◮ Representation theorem by Kallenberg for jointly exchangeable point processes on the plane ◮ Construction based on a completely random measure ◮ Properties of the model ◮ Exchangeability ◮ Sparsity ◮ Power-law degree distributions (with exponential cut-off) ◮ Interpretable parameters and hyperparameters ◮ Reinforced urn process construction ◮ Posterior characterization ◮ Scalable inference [Kallenberg, 2005, Caron and Fox, 2014] F. Caron 17 / 57

Point process representation ◮ Undirected graph represented as a point process on R 2 + � Z = z ij δ ( θ i ,θ j ) i,j with θ i ∈ R , z ij ∈ { 0 , 1 } with z ij = z ji 0 F. Caron 18 / 57

Point process representation Joint exchangeability Let A i = [ h ( i − 1) , hi ] for i ∈ N then d ( Z ( A i × A j )) = ( Z ( A π ( i ) × A π ( j ) )) for any permutation π of N and any h > 0 0 F. Caron 19 / 57

Point process representation ◮ Kallenberg derived a de Finetti style representation theorem for jointly and separately exchangeable point processes on the plane ◮ Representation via random transformations of unit rate Poisson processes and uniform variables ◮ Continuous-time equivalent of Aldous-Hoover for binary variables ◮ Our construction will fit into this framework [Kallenberg, 1990, Kallenberg, 2005] F. Caron 20 / 57

Completely random measures ◮ Nodes are embedded at some location θ i ∈ R + ◮ To each node is associated some sociability parameter w i ◮ Homogeneous completely random measure on R + ∞ � W = w i δ θ i W ∼ CRM ( ρ, λ ) . i =1 w i 0 θ i ◮ L´ evy measure ν ( dw, dθ ) = ρ ( dw ) λ ( dθ ) with λ the Lebesgue measure [Kingman, 1967] F. Caron 21 / 57

Completely random measures ◮ L´ evy measure ν ( dw, dθ ) = ρ ( dw ) λ ( dθ ) with λ the Lebesgue measure ◮ ρ is a measure on R + such that � ∞ (1 − e − w ) ρ ( dw ) < ∞ . (1) 0 which implies that W ([0 , T ]) < ∞ for any T < ∞ . � ∞ ρ ( dw ) = ∞ = ⇒ Infinite number of jumps in any interval [0 , T ] 0 “Infinite activity CRM” � ∞ ρ ( dw ) < ∞ = ⇒ Finite number of jumps in any interval [0 , T ] 0 “Finite activity CRM” F. Caron 22 / 57

Model for multi-edges directed graphs We represent the integer-weighted directed graph using an atomic measure on R 2 + ∞ ∞ � � D = n ij δ ( θ i ,θ j ) , i =1 j =1 where n ij counts the number of directed edges from node θ i to node θ j . 4 3 Counts 2 θ 2 θ 3 1 4 1 0 θ 2 θ 1 θ 2 2 3 θ 1 θ 1 θ 3 θ 3 F. Caron 23 / 57

Model for multi-edges directed graphs ◮ Conditional Poisson process with intensity measure � W = W × W on the product space R 2 + : D | W ∼ PP ( W × W ) 18 18 4.5 16 16 4 14 14 12 12 3.5 10 10 3 8 8 6 6 2.5 4 4 2 2 2 0 0 1.5 1 1 0.8 0.8 1 1 1 0.6 0.8 0.6 0.8 0.6 0.6 0.4 0.4 0.5 0.4 0.4 0.2 0.2 0.2 0.2 0 0 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 (d) Intensity measure � (c) CRM W W (e) Poisson process D F. Caron 24 / 57

Model for multi-edges directed graphs ◮ By construction, for any bounded intervals A and B of R + , � W ( A × B ) = W ( A ) W ( B ) < ∞ ◮ Finite number of counts over A × B ⊂ R 2 + D ( A × B ) < ∞ F. Caron 25 / 57

Model for undirected graphs ◮ Point process ∞ ∞ � � Z = z ij δ ( θ i ,θ j ) , i =1 j =1 with the convention z ij = z ji ∈ { 0 , 1 } ◮ Constructed from D by setting z ij = z ji = 1 if n ij + n ji > 0 and z ij = z ji = 0 otherwise 4 3 Counts 2 θ 2 1 θ 3 θ 2 θ 3 4 1 0 θ 2 θ 1 θ 1 θ 1 2 3 θ 2 θ 1 θ 3 θ 3 (a) D (b) Integer-valued (c) Undirected graph directed graph F. Caron 26 / 57

Model for undirected graphs ◮ Hierarchical model W = � ∞ i =1 w i δ θ i W ∼ CRM ( ρ, λ ) D = � ij n ij δ ( θ i ,θ j ) D ∼ PP ( W × W ) Z = � ij min( n ij + n ji , 1) δ ( θ i ,θ j ) F. Caron 27 / 57

Model for undirected graphs ◮ Equivalent direct formulation for i ≤ j � 1 − exp( − 2 w i w j ) i � = j Pr( z ij = 1 | w ) = 1 − exp( − w 2 i ) i = j and z ji = z ij 0 F. Caron 28 / 57

Outline Introduction Exchangeable matrices and their limitations Statistical network models using exchangeable random measures Exchangeability and sparsity properties Special case: Generalized gamma process Posterior characterization & Inference Experimental results F. Caron 29 / 57

Properties: Exchangeability Exchangeability Let h > 0 and A i = [ h ( i − 1) , hi ] , i ∈ N . By construction, d ( Z ( A i × A j )) = ( Z ( A π ( i ) × A π ( j ) )) for any permutation π of N . F. Caron 30 / 57

Properties: Sparsity ◮ W ( R + ) = ∞ , so infinite number of edges on R 2 + ◮ Restrictions D α and Z α of D and Z , respectively, to the box [0 , α ] 2 . ◮ N α number of nodes, and N ( e ) number of edges α α 0 α F. Caron 31 / 57

Properties: Sparsity N α N ( e ) 6 α 4 2 0 α 0 0 F. Caron 32 / 57