latent force models and multiple output gaussian processes
play

Latent Force Models and Multiple Output Gaussian Processes Neil D. - PowerPoint PPT Presentation

Dec 27, 2023 •153 likes •1.07k views

Latent Force Models and Multiple Output Gaussian Processes Neil D. Lawrence work with Magnus Rattray, Mauricio Alvarez, Pei Gao, Antti Honkela, David Luengo, Guido Sanguinetti, Michalis Titsias, Jennifer Withers SLIM Meeting 23rd July 2009

  1. Latent Force Models and Multiple Output Gaussian Processes Neil D. Lawrence work with Magnus Rattray, Mauricio Alvarez, Pei Gao, Antti Honkela, David Luengo, Guido Sanguinetti, Michalis Titsias, Jennifer Withers SLIM Meeting 23rd July 2009 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 1 / 36

  2. Outline Introduction 1 Latent Force Covariance Functions 2 Cascaded Differential Equations 3 Discussion and Future Work 4 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 2 / 36

  3. Outline Introduction 1 Latent Force Covariance Functions 2 Cascaded Differential Equations 3 Discussion and Future Work 4 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 3 / 36

  4. Dimensionality Reduction I Linear relationship between the data, X ∈ ℜ N × d , and a reduced dimensional representation, F ∈ ℜ N × q , where q ≪ d . X = FW + ǫ , ǫ ∼ N ( 0 , Σ ) Integrate out F , optimize with respect to W . For temporal data and a particular Gaussian prior in the latent space: Kalman filter/smoother. More generally consider a Gaussian process (GP) prior, q � � � p ( F | t ) = N f : , i | 0 , K f : , i , f : , i . i =1 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 4 / 36

  5. Dimensionality Reduction II Given the covariance functions for { f i ( t ) } the implied covariance functions for { x i ( t ) } — semi-parametric latent factor model (Teh et al., 2005). Linear Models of Coregionalization. Kalman filter/smoother approach has been preferred ◮ linear computational complexity in N . ◮ Advances in sparse approximations have made the general GP framework practical. (Snelson and Ghahramani, 2006; Qui˜ nonero Candela and Rasmussen, 2005; Titsias, 2009) . Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 5 / 36

  6. Mechanical Analogy These models rely on the latent variables to provide the dynamic information. We now introduce a further dynamical system with a mechanistic inspiration. Physical Interpretation: ◮ the latent functions, f i ( t ) are q forces. ◮ We observe the displacement of d springs to the forces., ◮ Interpret system as the force balance equation, XD = FS + ǫ . ◮ Forces act, e.g. through levers — a matrix of sensitivities, S ∈ ℜ q × d . ◮ Diagonal matrix of spring constants, D ∈ ℜ d × d . ◮ Original System: W = SD − 1 . Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 6 / 36

  7. Extend Model Add a damper and give the system mass. FS = ¨ XM + ˙ XC + XD + ǫ . Now have a second order mechanical system. It will exhibit inertia and resonance. There are many systems that can also be represented by differential equations. ◮ When being forced by latent function(s), { f i ( t ) } q i =1 , we call this a latent force model . Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 7 / 36

  8. Gaussian Process priors and Latent Force Models Driven Harmonic Oscillator For Gaussian process we can compute the covariance matrices for the output displacements. For one displacement the model is M � m k ¨ x k ( t ) + c k ˙ x k ( t ) + d k x k ( t ) = b k + s ik f i ( t ) , (1) i =0 where, m k is the k th diagonal element from M and similarly for c k and d k . s ik is the i , k th element of S . Model the latent forces as q independent, GPs with RBF covariances − ( t − t ′ ) 2 � � k f i f l ( t , t ′ ) = exp δ il . σ 2 i Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 8 / 36

  9. Covariance for ODE Model RBF Kernel function for f ( t ) � t q 1 � x j ( t ) = S ji exp( − α j t ) f i ( u ) exp( α j u ) sin( ω j ( t − u )) d u m j ω j 0 i =1 Joint distribution f(t) 0.8 for x 1 ( t ), x 2 ( t ), 0.6 y 1 (t) x 3 ( t ) and f ( t ). 0.4 Damping ratios: 0.2 y 2 (t) 0 ζ 1 ζ 2 ζ 3 −0.2 0.125 2 1 y 3 (t) −0.4 f(t) y 1 (t) y 2 (t) y 3 (t) Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 9 / 36

  10. Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

  11. Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

  12. Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

  13. Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

  14. Covariance for ODE RBF Kernel function for f ( t ) � t q 1 � x j ( t ) = S ji exp( − α j t ) f i ( u ) exp( α j u ) sin( ω j ( t − u )) d u m j ω j 0 i =1 Joint distribution f(t) 0.8 for x 1 ( t ), x 2 ( t ), 0.6 y 1 (t) x 3 ( t ) and f ( t ). 0.4 0.2 Damping ratios: y 2 (t) 0 ζ 1 ζ 2 ζ 3 −0.2 y 3 (t) 0.125 2 1 −0.4 f(t) y 1 (t) y 2 (t) y 3 (t) Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 11 / 36

  15. Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

  16. Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

  17. Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

  18. Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

  19. Example: Transcriptional Regulation r First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

  20. Example: Transcriptional Regulation r First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

  21. Example: Transcriptional Regulation r First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

Recommend

Latent Force Models with Gaussian Processes Neil D. Lawrence Bayesian Research Kitchen,

Latent Force Models with Gaussian Processes Neil D. Lawrence Bayesian Research Kitchen,

Latent Force Models with Gaussian Processes Neil D. Lawrence Bayesian Research Kitchen, Wordsworth Hotel, Grasmere 6th September 2008 Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 1 / 51 Outline Introduction 1

1.53k views • 98 slides
Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto

Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto

Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto University, Finland Why use S(P)DE solvers for GPs? The O ( n 3 ) computational complexity is always a challenge. Latent force models combine PDE/ODEs with

482 views • 12 slides
Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested

Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested

Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested Laplace Approximations BIOSTAT830: Graphical Models December 08, 2016 Introduction - INLA Inference for latent Gaussian Markov random field

605 views • 35 slides
E ffi cient Modeling of Latent Information in Supervised Learning using Gaussian Processes

E ffi cient Modeling of Latent Information in Supervised Learning using Gaussian Processes

E ffi cient Modeling of Latent Information in Supervised Learning using Gaussian Processes Mauricio A. Zhenwen Dai Alvarez Neil D. Lawrence Gaussian Process Approximation Workshop, 2017 Motivation I Machine learning has been very

685 views • 21 slides
Piecewise Bounds for Estimating Bernoulli- Logistic Latent Gaussian Models Mohammad Emtiyaz Khan

Piecewise Bounds for Estimating Bernoulli- Logistic Latent Gaussian Models Mohammad Emtiyaz Khan

Piecewise Bounds for Estimating Bernoulli- Logistic Latent Gaussian Models Mohammad Emtiyaz Khan Joint work with Benjamin Marlin, and Kevin Murphy University of British Columbia June 29, 2011 Piecewise Bounds for Binary Latent Gaussian Models

694 views • 34 slides
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models Yarin Gal Mark van der Wilk Carl E. Rasmussen yg279@cam.ac.uk June 25th, 2014 Outline Gaussian process regression and latent variable

1.18k views • 54 slides
Multiple-output Gaussian processes Mauricio A. Alvarez Department of Computer Science, The

Multiple-output Gaussian processes Mauricio A. Alvarez Department of Computer Science, The

Multiple-output Gaussian processes Mauricio A. Alvarez Department of Computer Science, The University of Sheffield. 1 / 76 Sensor Network South Coast of England Sensor location 2 / 76 Sensor Network South Coast of England Sensor

1.97k views • 192 slides
Latent Variable Models with Gaussian Processes Neil D. Lawrence GP Master Class 6th February

Latent Variable Models with Gaussian Processes Neil D. Lawrence GP Master Class 6th February

Latent Variable Models with Gaussian Processes Neil D. Lawrence GP Master Class 6th February 2017 Outline Motivating Example Linear Dimensionality Reduction Non-linear Dimensionality Reduction Outline Motivating Example Linear

973 views • 93 slides
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee

CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee

CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian Processes Outline Parametric

572 views • 40 slides
Probabilistic Graphical Models Lecture 21: Advanced Gaussian Processes Andrew Gordon Wilson

Probabilistic Graphical Models Lecture 21: Advanced Gaussian Processes Andrew Gordon Wilson

Probabilistic Graphical Models Lecture 21: Advanced Gaussian Processes Andrew Gordon Wilson www.cs.cmu.edu/~andrewgw Carnegie Mellon University April 1, 2015 1 / 100 Gaussian process review Definition A Gaussian process (GP) is a collection

1.36k views • 100 slides
1 Latent variable models In the next section we will discuss latent variable models for

1 Latent variable models In the next section we will discuss latent variable models for

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 16 notes: Latent variable models and EM Tues, 4.10 1 Latent variable models In the next section we will discuss latent

271 views • 4 slides
Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges in Unsupervised Learning Learn a latent variable model without labeled examples. E.g. topic models, hidden Markov models, Gaussian mixtures,

763 views • 61 slides
Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim

Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim

Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim 2 , Will Tebbutt 1 , J. Scott Hosking 3 , 4 , Arno Solin 5 , Richard E. Turner 1 , 6 1 University of Cambridge, 2 Invenia Labs, 3 British Antarctic

340 views • 23 slides
Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of

Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of

Approximative Bayesian inference Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of Mathematical Sciences NTNU, Norway December 4, 2009 1 With S.Martino/N.Chopin Approximative Bayesian inference Overview

1.61k views • 159 slides
Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian posteriors Approximate methods Laplace approximation Variational bayes Expectation propagation Comparisons GP regression - recap so far Model the

1.41k views • 113 slides
Learning unknown forces in nonlinear models with Gaussian processes and autoregressive flows Wil

Learning unknown forces in nonlinear models with Gaussian processes and autoregressive flows Wil

Learning unknown forces in nonlinear models with Gaussian processes and autoregressive flows Wil O C Ward w.ward@sheffield.ac.uk Department of Physics and Astronomy, The University of Sheffield GPSS Workshop: Structurally Constrained Gaussian

923 views • 89 slides
Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes Pierre-Franois Lget, Stanford University - KIPAC Gaussian process (GP) interpolation: Gaussian processes are a non-parametric way to interpolate data

281 views • 7 slides
Prior Knowledge and Sparse Methods for Convolved Multiple Outputs Gaussian Processes Mauricio A.

Prior Knowledge and Sparse Methods for Convolved Multiple Outputs Gaussian Processes Mauricio A.

Prior Knowledge and Sparse Methods for Convolved Multiple Outputs Gaussian Processes Mauricio A. lvarez Joint work with Neil D. Lawrence, David Luengo and Michalis K. Titsias School of Computer Science University of Manchester (University of

826 views • 78 slides
CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian Processes Fancier Gaussian Processes The current DFF. ( de facto fanciness) Uses for: Regression Classification

754 views • 50 slides
Identifiability of Gaussian DAG models with one latent source Hisayuki Hara Niigata University

Identifiability of Gaussian DAG models with one latent source Hisayuki Hara Niigata University

Identifiability of Gaussian DAG models with one latent source Hisayuki Hara Niigata University http://www.econ.niigata-u.ac.jp/hara/ hara@econ.niigata-u.ac.jp Joint work with Dennis Leung and Mathias Drton H. Hara (Niigata U.)

527 views • 26 slides
State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2 Alexander Grigorievskiy 2 , 3 1 Philips Research, 2 Aalto University, 3 Silo.AI ICML2018 July 13, 2018 Outline Gaussian Processes Temporal GPs as

285 views • 17 slides
Latent Force Models Neil D. Lawrence (work with Magnus Rattray, Mauricio Alvarez , Pei Gao,

Latent Force Models Neil D. Lawrence (work with Magnus Rattray, Mauricio Alvarez , Pei Gao,

Latent Force Models Neil D. Lawrence (work with Magnus Rattray, Mauricio Alvarez , Pei Gao, Antti Honkela, David Luengo, Guido Sanguinetti, Michalis Titsias, Jennifer Withers) University of Sheffield University of Edinburgh Bayes 250

625 views • 43 slides
Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at

Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at

Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at CVPR 2012 Erik Sudderth Brown University NIPS 2008: E. Sudderth & M. Jordan, Shared Segmentation of Natural Scenes using Dependent Pitman-Yor

814 views • 48 slides
Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics University of Sheffield GP summer school September 2017 Why use Gaussian processes? Why use Gaussian processes? A stochastic process is a collection

1.69k views • 152 slides

More recommend