Latent Force Models with Gaussian Processes Neil D. Lawrence - PowerPoint PPT Presentation

Latent Force Models with Gaussian Processes Neil D. Lawrence Bayesian Research Kitchen, Wordsworth Hotel, Grasmere 6th September 2008 Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 1 / 51

Outline Introduction 1 Covariance Functions 2 Convolutions and Computational Complexity 3 Non-linear Response Models 4 Cascaded Differential Equations 5 Discussion and Future Work 6 Acknowledgements 7 Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 2 / 51

Outline Introduction 1 Covariance Functions 2 Convolutions and Computational Complexity 3 Non-linear Response Models 4 Cascaded Differential Equations 5 Discussion and Future Work 6 Acknowledgements 7 Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 3 / 51

Dimensionality Reduction I Linear relationship between the data, X ∈ ℜ N × d , and a reduced dimensional representation, F ∈ ℜ N × q , where q ≪ d . X = FW + ǫ , ǫ ∼ N ( 0 , Σ ) Integrate out X , optimize with respect to W . For temporal data and a particular Gaussian prior in the latent space: Kalman filter/smoother More generally consider a Gaussian process (GP) prior, q � � � p ( F | t ) = N f : , i | 0 , K f : , i , f : , i . i =1 Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 4 / 51

Dimensionality Reduction II Given the covariance functions for { f i ( t ) } the implied covariance functions for { x i ( t ) } — semi-parametric latent factor model (Teh et al., 2005). Kalman filter/smoother approach has been preferred ◮ linear computational complexity in N . ◮ Advances in sparse approximations have made the general GP framework practical. (Snelson and Ghahramani, 2006; Qui˜ nonero Candela and Rasmussen, 2005, also Titsias tomorrow) . Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 5 / 51

Mechanical Analogy These models rely on the latent variables to provide the dynamic information. We now introduce a further dynamical system with a mechanistic inspiration. Physical Interpretation: ◮ the latent functions, f i ( t ) are q forces. ◮ We observe the displacement of d springs to the forces., ◮ Interpret system as the force balance equation, XD = FS ǫ . ◮ Forces act, e.g. through levers — a matrix of sensitivities, S ∈ ℜ q × d . ◮ Diagonal matrix of spring constants, D ∈ ℜ d × d . ◮ Original System: W = SD − 1 . Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 6 / 51

Extend Model Add a damper and give the system mass. FS = ¨ XM + ˙ XC + XD + ǫ . Now have a second order mechanical system. It will exhibit inertia and resonance. There are many systems that can also be represented by differential equations. ◮ When being forced by latent function(s), { f i ( t ) } q i =1 , we call this a latent force model . Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 7 / 51

Gaussian Process priors and Latent Force Models For Gaussian process we can compute the covariance matrices for the output displacements. For one displace the model is M � m k ¨ x k ( t ) + c k ˙ x k ( t ) + d k x k ( t ) = b k + s ik f i ( t ) , (1) i =0 where, m k is the k th diagonal element from M and similarly for c k and d k . s ik is the i , k th element of S . Model the latent forces as q independent, GPs with RBF covariances − ( t − t ′ ) 2 � � k f i f l ( t , t ′ ) = exp δ il . σ 2 i Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 8 / 51

Covariance for ODE Model RBF Kernel function for f ( t ) q � t 1 � x j ( t ) = S ji exp( − α j t ) f i ( u ) exp( α j u ) sin( ω j ( t − u )) d u m j ω j 0 i =1 Joint distribution f(t) 0.8 for x 1 ( t ), x 2 ( t ), 0.6 y 1 (t) x 3 ( t ) and f ( t ). 0.4 0.2 Damping ratios: y 2 (t) 0 ζ 1 ζ 2 ζ 3 −0.2 y 3 (t) 0.125 2 1 −0.4 f(t) y 1 (t) y 2 (t) y 3 (t) Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 9 / 51

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

Covariance for ODE RBF Kernel function for f ( t ) q � t 1 � x j ( t ) = S ji exp( − α j t ) f i ( u ) exp( α j u ) sin( ω j ( t − u )) d u m j ω j 0 i =1 Joint distribution f(t) 0.8 for x 1 ( t ), x 2 ( t ), 0.6 y 1 (t) x 3 ( t ) and f ( t ). 0.4 0.2 Damping ratios: y 2 (t) 0 ζ 1 ζ 2 ζ 3 −0.2 y 3 (t) 0.125 2 1 −0.4 f(t) y 1 (t) y 2 (t) y 3 (t) Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 11 / 51

Example: Transcriptional Regulation First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

Example: Transcriptional Regulation First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

Example: Transcriptional Regulation First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

Example: Transcriptional Regulation First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

Example: Transcriptional Regulation First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

Example: Transcriptional Regulation First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51