Exponential Random Graph Models and Their Polytopes Johannes Rauh - PowerPoint PPT Presentation

Exponential Random Graph Models and Their Polytopes Johannes Rauh York University (the one in Canada) AMS Sectional Meeting Chicago 2015 J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 1 / 16

Outline Discrete Exponential Families 1 Graphical Models 2 Exponential Random Graph Models 3 J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 2 / 16

Discrete Exponential Families Given a finite sample space X and a vector of statistics f : X → R h , we can associate a discrete exponential family P θ : P θ ( x ) = 1 � � E = exp( � θ, f x � ) Z θ Theorem � H ( P ) � . Let m = E P θ [ f ] . Then P θ = argmax P : E P [ f ] = m Jaynes’ principle of maximum entropy: If you know nothing about a distribution but its expectation value E [ f ] , you should use P θ . Theorem (MLE) Let P be the empirical distribution of some data set, and suppose that E P [ f ] = E P θ [ f ] . Then P θ is the unique maximum likelihood estimate (MLE). J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 3 / 16

The Moment Map P θ ( x ) > 0 for all x , θ . However, P ( x ) = 0 is possible in the closure/boundary of E . The closure is denoted by E . Definition The map µ : P �→ E P [ f ] is the moment map. The image M = conv � f x : x ∈ X � . is the convex support polytope Theorem µ restricts to a bijection E � M . The inverse will be denoted by µ − 1 . Theorem Suppose m belongs to the face F ⊆ M , and let P = µ − 1 ( m ) . Then P ( x ) > 0 if and only if f x ∈ F . J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 4 / 16

Example The independence model of two binary variables: The idea is that E “looks like” M . The combinatorics of M reflect properties of E . J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 5 / 16

Generalized MLEs Theorem (Generalized MLE) Let P be the empirical distribution, and let m = E P [ f ] . Then µ − 1 ( m ) is the unique maximum likelihood estimate (MLE) within E . The MLE within E is also called the generalized MLE (GMLE) for E . If µ − 1 ( m ) belongs to the boundary ∂ E : = E \ E , one says that “the MLE does not exist.” If the MLE does not exist: not enough data? 1 structural zero? 2 The GMLE has no parameters, so how do you estimate? J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 6 / 16

Graphical Models Let G = ( V , E ) be a graph with n = | V | nodes. To each node v ∈ V associate a finite random variable X v , taking values in X v . Thus, X = × v ∈ V X v . A joint distribution of ( X v ) v ∈ V is a | V | -dimensional tensor ( p x 1 , x 2 ,..., x n ) x 1 ∈X 1 , x 2 ∈X 2 ,..., x n ∈X n . Denote by C ( G ) the set of cliques of G (i.e. the complete subgraphs). Definition The graphical model is the set of all probability distributions P on X of the form � P ( x 1 , . . . , x n ) = φ C ( x i 1 , . . . , i k ) , C = { i 1 ,..., i k }∈C ( G ) where φ C is a positive function. Lemma The graphical model is a discrete exponential family. The vector of statistics contains the C -marginals for all C ∈ C ( G ) . J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 7 / 16

Marginal Polytopes The convex support M of a graphical model is a marginal polytope. Marginal polytopes are 0/1-polytopes (each f x is a 0/1-vector) Therefore, each f x is a vertex of M . Moreover, M is a subpolytope of a hypercube. Every f x has the same number of ones (in some parametrization. . . ) Therefore, all f x lie on a sphere. Marginal polytopes are symmetric (the symmetry group acts transitively on the vertices) Graphical models have been proven to be a versatile modelling platform in many applications. J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 8 / 16

Exponential Random Graph Models Exponential Random Graph Models (ERGMs) are discrete exponential families where the sample space G is a set of graphs, e.g. G = G n : = � graphs on n nodes � . Any choice of graph statistics defines an ERGM, for example: Subgraph counts (triangles, cycles, k -stars, . . . ) Degree statistics (average degree, degree distribution, degree sequence, . . . ) J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 9 / 16

Changing the Number of Nodes Often, the number of nodes n is fixed in theory; i.e. the sample space is G = G n : = � graphs on n nodes � . In applications it is often a variable. What happens when changing n ? Many graph statistics behave “continuously” when changing n . However: Need to take into account scaling of the parameters! For subgraph densities (normalized counts), the convex support polytopes converge (Engstr¨ en 2011) : om, Nor´ � M n ⊇ M n + 1 ⊇ · · · ⊇ M ∞ : = n ′ M n ′ . For other statistics (e.g. degree sequences), the number of parameters depends on n (Chatterjee, Diaconis 2011) n plays a role similar to the sample size. sparse vs. dense graphs: Restrict G or penalize large edge density? J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 10 / 16

ERGMs and Small Samples When analyzing networks, often there is just a single observation. The MLE might still exist—provided most f G are not vertices. For many ERGMs, this is indeed the case. The “geometry” of such an ERGMs is determined not only by their convex support M , but also by the location of the points f G within M . Example: Edges and 2-stars (Rauh 2012) n = 6 n = 7 1 1 0.5 0.5 0 0 0 0.5 1 0 0.5 1 J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 11 / 16

An LDP for ERGMs Chatterjee and Varadhan (2010) proved a Large Deviation Principle that describes the distribution of f G for many statistics for large n . Intuition: The uniform distribution on (labelled!) graphs on n nodes is � n � equivalent to independent random variables. 2 As a consequence, most f G lie close to the center of mass of M (corresponding to the uniform distribution). Of those f G that don’t lie close to the center of mass, most lies close to some Erd˝ os-R´ enyi graph. J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 12 / 16

Asymptotics of ERGMs and Degeneracy Diaconis and Chatterjee (2011) showed for certain examples of subgraph counts that for large n and “reasonably scaled parameters” θ , the random graph P θ lies close to some Erd˝ os-R´ enyi graph. In principle, you can always escape the Erd˝ os-R´ enyi by choosing large parameters—but maybe not in all directions! n = 7 1 0.5 0 0 0.5 1 A similar degeneracy has been observed in applications: The MLEs of real networks are often close to Erd˝ os-R´ enyi. = ⇒ If you are not happy with Erd˝ os-R´ enyi, you need to change f . J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 13 / 16

Degree Sequences and Partitions Some ERGMs don’t have interior points: Let f ds ( G ) = the degree sequence. M ds many interior points. Let f dp ( G ) = the degree sequence ordered by magnitude, larger degrees first (degree partition). M dp has no interior point. (reason: every degree partition has a repetition d i = d i + 1 , and d i ≥ d i + 1 defines a facet) Let f dd ( G ) = the degree distribution ( � all k -stars). M dd has no interior point. (reason: a graph cannot have both an isolated node and a fully connected node) M dp and M ds are related (Bhattacharya, Sivasubramanian, Srinivasan 2006) : M ds consists of n ! copies of M dp , corresponding to all ways of ordering the nodes. Further example (joint work with K. Sadeghi, T. Short, ´ E. Czabarka, L. Szekely) : Bi-degree statistics: Find the smallest face containing some f G in its interior. J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 14 / 16

So. . . what’s the difference between graphical models and ERGMs? J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 15 / 16

So. . . what’s the difference between graphical models and ERGMs? In graphical models, the nodes are random, but in ERGMS, the edges are random. J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 15 / 16

So. . . what’s the difference between graphical models and ERGMs? In graphical models, the nodes are random, but in ERGMS, the edges are random. Technically true, but. . . J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 15 / 16

Comparing graphical models and ERGMs Both are very different examples of discrete exponential families. Graphical models are a fixed (but very flexible) class of models. Properties of ERGMs depend very much on the chosen statistics. Typically: graphical models ERGMs every f x is a vertex ←→ most f G are not vertices f x distributed on a sphere ←→ f G cluster at the center of mass high symmetry ←→ almost no symmetry J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 16 / 16

Comparing graphical models and ERGMs Both are very different examples of discrete exponential families. Graphical models are a fixed (but very flexible) class of models. Properties of ERGMs depend very much on the chosen statistics. Typically: graphical models ERGMs every f x is a vertex ←→ most f G are not vertices f x distributed on a sphere ←→ f G cluster at the center of mass high symmetry ←→ almost no symmetry Actually, there is a way to combine ERGMs and graphical models: Study conditional independences between edges (Lauritzen, Rinaldo, Sadeghi) J. Rauh (YorkU) Exponential Random Graph Models AMS 2015 16 / 16