Categorical models of probability with symmetries Sam Staton, - PowerPoint PPT Presentation

Categorical models 
 of probability 
 with symmetries Sam Staton, Oxford

Categorical models 
 of probability with symmetries My starting point: • Probabilistic programming is an internal language for categorical probability theory 
 (as well as a useful practical tool in stats/ML). • Programming languages already have mechanisms for abstraction and invariance. • These can give new perspectives on 
 symmetry in probability.

Plan of talk: 1. Intuitive illustrations of symmetries in a. random graphs b. beta distributions / Pólya urns 2. Models for a. beta / Pólya urns Staton, Stein, Yang, Ackerman, Freer, Roy, ICALP 2018. b. random graphs ongoing work with Ackerman, Freer, Roy, Yang.

Building infinite random graphs a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 && not(edge(b,c)))

Building infinite random graphs a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 && not(edge(b,c))) a b c

Building infinite random graphs a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Example: && not(edge(b,c))) get() = uniform S n edge(p,q) 
 = if d(p,q) < π / 2 
 then True else False

Building infinite random graphs Interface: 
 a <- get() b <- get() get() : node 
 return (edge(a,b)) edge(node,node) : bool Example: get() = uniform S n edge(p,q) 
 = if d(p,q) < π / 2 
 then True else False

Building infinite random graphs a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Example: && not(edge(b,c))) get() = uniform S n edge(p,q) 
 = if d(p,q) < π / 2 
 then True else False

Building infinite random graphs a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Data flow graph && not(edge(b,c))) a get b get horn? get c

Building infinite random graphs c <- get() Interface: 
 b <- get() get() : node 
 a <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Data flow graph && not(edge(b,c))) a get b get horn? get c

Building infinite random graphs a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(c,b) && edge(c,a) 
 Data flow graph && not(edge(b,a))) c get b get horn? get a Invariance under symmetries of data flow = graph exchangeability

Building infinite random graphs Interface: 
 get() : node 
 edge(node,node) : bool a <- get() The b <- get() interface return (a < b) doesn’t a <- get() allow: b <- get() return (sin(a) = cos(b))

Building infinite random graphs Interface: 
 Invariance under changes to implementation 
 get() : node 
 = graph exchangeability edge(node,node) : bool a <- get() The b <- get() interface return (a < b) doesn’t a <- get() allow: b <- get() return (sin(a) = cos(b))

Summary of symmetries Interface: 
 get() : node 
 edge(node,node) : bool Invariance under implementation details 
 + data flow symmetries 
 = 
 graph exchangeability 
 (Aldous-Hoover)

Another model a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Example: && not(edge(b,c))) get() = uniform S n edge(p,q) = [d(p,q) < π / 2 ] building on 
 Bubeck, Ding, Eldan, Racz, 2015 
 Devroye, György, Lugosi, Udina, 2011

Another model a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Example: && not(edge(b,c))) get() = uniform(0,1) edge(p,q) 
 = memoize p,q (bernoulli(0.5)) building on 
 Bubeck, Ding, Eldan, Racz, 2015 
 Devroye, György, Lugosi, Udina, 2011

Another model a <- get() Interface: 
 b <- get() get() : node 
 c <- get() edge(node,node) : bool return (edge(a,b) && edge(a,c) 
 Example: && not(edge(b,c))) get() = uniform(0,1) edge(p,q) 
 = memoize p,q (bernoulli(0.5)) Roy, a Mansinghka, get Goodman, Tenenbaum, ICML 2008 b get horn? building on 
 Bubeck, Ding, Eldan, Racz, 2015 
 get c Devroye, György, Lugosi, Udina, 2011

Summary of symmetries Interface: 
 get() : node 
 edge(node,node) : bool Invariance under implementation details 
 + data flow symmetries 
 = 
 graph exchangeability 
 (Aldous-Hoover)

Plan of talk: 1. Intuitive illustrations of symmetries in a. random graphs b. beta distributions / Pólya urns 2. Models for a. beta / Pólya urns Staton, Stein, Yang, Ackerman, Freer, Roy, ICALP 2018. b. random graphs ongoing work with Ackerman, Freer, Roy, Yang.

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2))

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2)) b1 sample get -&not(-) a sample b2

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2)) Example: get() = uniform(0,1) sample(p) = bernoulli(p)

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2)) Example: get() = uniform(0,1) Prob(return True) = sample(p) = bernoulli(p) 1 ∫ p (1 − p ) d p = 1 6 0

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2)) Example: After get() = uniform(0,1) 10000 sample(p) = bernoulli(p) runs… 1632 8368 True False

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2)) Example: get() = uniform(0,1) sample(p) = bernoulli(p)

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- uniform(0,1) b1 <- bernoulli(a) get() : I 
 b2 <- bernoulli(a) sample(I) : bool return (b1 & not(b2)) Example: get() = uniform(0,1) sample(p) = bernoulli(p)

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- uniform(0,1) b1 <- bernoulli(a) get() : I 
 b2 <- bernoulli(a) sample(I) : bool return (b1 & not(b2)) Probability density a

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- uniform(0,1) b1 <- bernoulli(a) get() : I 
 b2 <- bernoulli(a) sample(I) : bool return (b1 & not(b2)) Probability density 1 b1 <- bernoulli( ) beta(1,2) beta(2,1) 2 a <- beta(1+b1,2-b1) b2 <- bernoulli(a) return (b1 & not(b2)) a

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- uniform(0,1) b1 <- bernoulli(a) get() : I 
 b2 <- bernoulli(a) sample(I) : bool return (b1 & not(b2)) Probability density beta(1,3) beta(3,1) 1 b1 <- bernoulli( ) beta(1,2) beta(2,1) 2 beta(2,2) 1 + 𝚌𝟸 b2 <- bernoulli( ) 3 a <- beta( 1+b1+b2,3-b1-b2 ) a return (b1 & not(b2))

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- uniform(0,1) b1 <- bernoulli(a) get() : I 
 b2 <- bernoulli(a) sample(I) : bool return (b1 & not(b2)) Probability density beta(1,3) beta(3,1) 1 b1 <- bernoulli( ) beta(1,2) beta(2,1) 2 beta(2,2) 1 + 𝚌𝟸 b2 <- bernoulli( ) 3 a <- beta( 1+b1+b2,3-b1-b2 ) a return (b1 & not(b2))

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- uniform(0,1) b1 <- bernoulli(a) get() : I 
 b2 <- bernoulli(a) sample(I) : bool return (b1 & not(b2)) Prob(return True) = 1 b1 <- bernoulli( ) 1 2 ⋅ 1 3 = 1 2 1 + 𝚌𝟸 6 b2 <- bernoulli( ) 3 a <- beta( 1+b1+b2,3-b1-b2 ) No integration required! return (b1 & not(b2))

Example 2: Beta / bernoulli Given an unknown coin, what is the probability of 
 heads then tails? Interface: 
 a <- get() b1 <- sample(a) get() : I 
 b2 <- sample(a) sample(I) : bool return (b1 & not(b2)) Another example: get() = new urn sample(p) = Pólya draw: one out, two in