Outline 1. Motivation 2. Gaussian process introduction 3. Change - PDF document

5/22/2016 Scalable Gaussian Processes for Characterizing Multidimensional Change Surfaces April 18, 2016 William Herlands Committee: Daniel Neill, Alex Smola, Wilbert Van Panhuis Chair: Dave Choi Outline 1. Motivation 2. Gaussian process introduction 3. Change surface model 4. Analysis of measles in the United States 1

5/22/2016 Complex Changes • In human systems changes are often complex – Policy interventions take time to trickle through government bureaucracy – Environmental hazards affect populations differentially • Simple changepoint models are not sufficiently expressive Why do we care? • Understand past changes – Explore spatio-temporal heterogeneity – Model the rate of changes in different areas • Enable more accurate or equitable policies • Applications – Measles incidence in the U.S – Concerns about lead-tainted 08 Jul 2014 21 Oct 2015 water in NYC 2

5/22/2016 Our objectives • Model complex changes in real world data – Multiple, flexible function Gaussian processes regimes for flexible functions – Non-discrete changes – Non-monotonic changes “Change surfaces” for complex changes – Heterogeneous changes over space, time, etc. Gaussian Processes (GP) • Non-parametric prior over smooth functions f ( x ) ~ GP ( m ( x ), k ( x , x ')) m ( x )  E [ f ( x )] k ( x , x ')  cov( f ( x ), f ( x ')) • Covariance function is a kernel. Defines the covariance of function values 3

5/22/2016 Gaussian Processes (GP) • Any finite set of f( x ) is Normally distributed   ~ N ( m ( x ), K ) f ( x 1 ),..., f ( x m ) • Observation model  ~ N (0,   ) y ( x )  f ( x )   , • Marginal log likelihood optimization log p ( y |  )  log | K    I |  y T ( K    I )  1 y Full Model • Our model is a convex combination of f i __ __ __ __ y ( x )  s 1 ( x ) f 1 ( x )  ...  s r ( x ) f r ( x )   n Switching functions Functional regimes i ( x )   r s 4

5/22/2016 Model part 1: Functional Regimes • GP prior for each functional regime – Use flexible stationary kernels f i ~ GP (0, K i ), i  1,..., r Model part 2: Change Surfaces • Changepoint 1 i  I ( t  T s i ) 0.5 0 −10 0 10 • Non-discrete changepoint 1 i  softmax( t  T s i ) 0.5 0 • Change surface −10 0 10 i  softmax( w 1 s i ( t )) 0.5 i   ( w i ( t )) s 0 −10 0 10 5

5/22/2016 Model part 2: Change Surfaces w i ( x ) • Random Kitchen Sink features for – Variable rate of change – Non-monotonic – Heterogeneous over input q  w i ( x )  cos(  j T x  b j ) a j j  1 Full Model • Gaussian process change surface model r  y ( x )   ( w i ( x )) f i ( x )   n ______ i  1 f i ( x ) ~ GP (0, K i ) • Can depict this as a single Gaussian process with covariance function r  k all ( x , x ')   ( w i ( x )) k i ( x , x ')  ( w i ( x ')) i  1 6

5/22/2016 Scalable Inference • Log likelihood naively O(n 3 ) log p ( y |  )  log | K    I |  y T ( K    I )  1 y • We develop scalable Kronecker inference using the Weyl bound, O(Dn D+1/D ) Measles in the United States • Data – Monthly incidence rates 1935 – 2003 – Continental United States and D.C. x   3 , 2D space and 1D time – – Measles vaccine introduced in 1963 7

5/22/2016 Measles in 3 states California 1 300 Incidence (1000s) CA 200 s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Maine 1 200 Incidence (1000s) ME s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Michigan 500 1 Incidence (1000s) 400 MI 300 s(w(x)) 0.5 200 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Measles in 3 states California 1 300 Incidence (1000s) CA 200 s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Maine 1 200 Incidence (1000s) ME s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Michigan 500 1 Incidence (1000s) 400 MI 300 s(w(x)) 0.5 200 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 8

5/22/2016 Measles in 3 states • GP change surface – 2 functional regimes w i ( x ) – as RKS with 5 features • Not a causal model! Measles in 3 states California 300 1 Incidence (1000s) CA 200 s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Maine 1 200 Incidence (1000s) ME s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Michigan 500 1 Incidence (1000s) 400 MI 300 s(w(x)) 0.5 200 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 9

5/22/2016 Measles in 3 states California 300 1 Incidence (1000s) CA 200 s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Maine 1 200 Incidence (1000s) ME s(w(x)) 0.5 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Michigan 500 1 Incidence (1000s) 400 MI 300 s(w(x)) 0.5 200 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 Measles in 3 states “Change slope” from σ(w(x)) = 0.25  0.75 . Michigan 500 1 Incidence (1000s) 400 “Change date” per state 300 s(w(x)) MI 0.5 σ(w(x)) = 0.5 200 100 0 0 1940 1940 1950 1950 1960 1960 1970 1970 1980 1980 1990 1990 2000 2000 2010 2010 10

5/22/2016 Change date for measles in U.S. 1961.5 1967.2 For each state, date where σ(w(x)) = 0.5 Change slope for measles in U.S. 0.156 0.297 For each state, slope of σ(w(x)) = 0.75  0.25 11

5/22/2016 Regression Analysis • Explore factors that affect the change date – Birth and death rates – Population numbers per age segment – Income information – Government hospital and health workers – Slope of change surface – Average temperature Demographic Analysis 12

5/22/2016 Regression Analysis • Gini of family income 1961.5 1967.2 – Economically depressed communities – Rural regions • Slope of change surface – Fewer cases nationwide enable more effective immunization later Conclusions • Introduced model for “change surfaces” in real world data • Developed scalable inference for additive, non-stationary Gaussian processes • Identified heterogeneity in first years of the measles vaccine • Used the results of the change surface model for policy relevant conclusions 13

5/22/2016 Acknowledgements • Committee – Daniel Neill, Alex Smola, Wilbert van Panhuis • Chair – Dave Choi • Collaborators* – Andrew Wilson – Seth Flaxman – Hannes Nickisch *Subset of paper accepted to AISTATS 2016 Questions? Fin. 28 14

5/22/2016 Backup slides Conclusions • Introduced model for “change surfaces” in real world data • Developed scalable inference for additive, non-stationary Gaussian processes • Identified heterogeneity in first years of the measles vaccine • Used the results of the change surface model for policy relevant conclusions 15

5/22/2016 Spectral Mixture Kernels Inference • Compute log marginal likelihood • General Kronecker methods for scalability – Assume: – Assume: multiplicative kernel across D – Then we can decompose kernel matrix, 16

5/22/2016 Inference • For additive kernels • K -1 can be computed efficiently using LCG* • But how can we compute the log|K| ? *See Flaxman et al. (2015) Inference 17

5/22/2016 Inference • Choosing indices i, j Method Complexity Minimization for best pair O(n 2 ) “Middle” heuristic i=j O(n) OR i=j+1 Greedy search of s pairs O(2sn) below and above previous pair Inference • Scaling functions, σ(w(x)) 18

5/22/2016 Inference 3 Kernels 3 Kernels 4 10 1 Weyl exact 0.8 Weyl middle Log determinant approximation ratio Weyl greedy 0.6 Cheb−Hutch 2 10 True log det 0.4 0.2 Time (sec) 0 10 0 −0.2 −0.4 −2 10 −0.6 −0.8 −4 −1 10 2 3 4 2 4 6 10 10 10 10 10 10 Observations (#) Observations (#) Inference – so what?! • Linear complexity for additive kernels – O(Dn D+1/D ) • Scalable inference for non-separable kernels in space and time • Scalable inference for non-stationary kernels 19

5/22/2016 Numerical Experiments • 2500 points of synthetic data • 2 functional regimes defined by squared exponential kernels • Change surface define by Results - Numerical 20

5/22/2016 Demographic Analysis Demographic Analysis 21