a unifying framework for sparse gaussian process
play

A Unifying Framework for Sparse Gaussian Process Approximation using - PowerPoint PPT Presentation

Feb 08, 2024 •155 likes •884 views

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

  1. A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint work with Thang Bui, Cuong Nguyen and Josiah Yan 1 / 22

  2. Motivation: Gaussian Process Regression outputs inputs 2 / 22

  3. Motivation: Gaussian Process Regression ? outputs inputs 2 / 22

  4. Motivation: Gaussian Process Regression ? outputs inputs 2 / 22

  5. Motivation: Gaussian Process Regression inference & learning ? outputs inputs 2 / 22

  6. Motivation: Gaussian Process Regression inference & learning intractabilities computational analytic ? outputs inputs 2 / 22

  7. A Brief History of Gaussian Process Approximations FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  8. A Brief History of Gaussian Process Approximations approximate generative model exact inference FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  9. A Brief History of Gaussian Process Approximations approximate generative model methods employing exact inference pseudo-data FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  10. A Brief History of Gaussian Process Approximations approximate generative model methods employing exact inference pseudo-data FITC PITC DTC FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  11. A Brief History of Gaussian Process Approximations approximate generative model methods employing exact inference pseudo-data A Unifying View of Sparse Approximate Gaussian Process Regression FITC Quinonero-Candela & PITC Rasmussen, 2005 (FITC, PITC, DTC) DTC FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  12. A Brief History of Gaussian Process Approximations approximate generative model exact generative model methods employing exact inference approximate inference pseudo-data A Unifying View of Sparse Approximate Gaussian Process Regression FITC Quinonero-Candela & PITC Rasmussen, 2005 (FITC, PITC, DTC) DTC FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  13. A Brief History of Gaussian Process Approximations approximate generative model exact generative model methods employing exact inference approximate inference pseudo-data A Unifying View of Sparse Approximate Gaussian Process Regression FITC VFE Quinonero-Candela & PITC EP Rasmussen, 2005 (FITC, PITC, DTC) DTC PP FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  14. A Brief History of Gaussian Process Approximations approximate generative model exact generative model methods employing exact inference approximate inference pseudo-data A Unifying View of Sparse Approximate Gaussian Process Regression FITC VFE Quinonero-Candela & PITC EP Rasmussen, 2005 (FITC, PITC, DTC) DTC PP FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  15. A Brief History of Gaussian Process Approximations approximate generative model exact generative model methods employing exact inference approximate inference pseudo-data A Unifying View of Sparse Approximate Gaussian Process Regression FITC VFE Quinonero-Candela & PITC EP Rasmussen, 2005 (FITC, PITC, DTC) DTC PP A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Bui, Yan and Turner, 2016 (VFE, EP, FITC, PITC ...) FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression” 3 / 22

  16. EP pseudo-point approximation true posterior 4 / 22

  17. EP pseudo-point approximation true posterior 4 / 22

  18. EP pseudo-point approximation marginal posterior likelihood true posterior 4 / 22

  19. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 4 / 22

  20. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 4 / 22

  21. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 4 / 22

  22. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 4 / 22

  23. EP pseudo-point approximation exact joint of new GP regression model marginal posterior likelihood true posterior approximate posterior input locations of outputs and covariance 'pseudo' data 'pseudo' data 4 / 22

  24. EP algorithm 5 / 22

  25. EP algorithm take out one 1. remove pseudo-observation likelihood cavity 5 / 22

  26. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood tilted 5 / 22

  27. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 5 / 22

  28. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family update 4. update pseudo-observation likelihood 5 / 22

  29. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 1. minimum: moments matched at pseudo-inputs 2. Gaussian regression: matches moments everywhere update 4. update pseudo-observation likelihood 5 / 22

Recommend

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

715 views • 68 slides
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models Yarin Gal Mark van der Wilk Carl E. Rasmussen yg279@cam.ac.uk June 25th, 2014 Outline Gaussian process regression and latent variable

1.18k views • 54 slides
Variational Model Selection for Sparse Gaussian Process Regression Michalis K. Titsias School of

Variational Model Selection for Sparse Gaussian Process Regression Michalis K. Titsias School of

Variational Model Selection for Sparse Gaussian Process Regression Variational Model Selection for Sparse Gaussian Process Regression Michalis K. Titsias School of Computer Science University of Manchester 7 September 2008 Variational Model

1.05k views • 41 slides
Sparse Gaussian Process Approximations Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and

Sparse Gaussian Process Approximations Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and

Sparse Gaussian Process Approximations Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge 1 / 90 Motivating application 1: Audio modelling audio 5

1.54k views • 119 slides
Gaussian Process Lei Tang Arizona State University Jul. 31th, 2007 Lei Tang (ASU) Gaussian

Gaussian Process Lei Tang Arizona State University Jul. 31th, 2007 Lei Tang (ASU) Gaussian

Gaussian Process Lei Tang Arizona State University Jul. 31th, 2007 Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 1 / 22 Gaussian Process, (kriging in geostatistics) Autoregressive moving average model, Kalman filters, and radial basis

340 views • 31 slides
Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression Predictions using noisy observations The case of a single test input: Gaussian Process Regression Computational and numerical issues It is

444 views • 29 slides
CS 240A: Solving Ax = b in parallel Dense A: Gaussian elimination with partial pivoting (LU)

CS 240A: Solving Ax = b in parallel Dense A: Gaussian elimination with partial pivoting (LU)

CS 240A: Solving Ax = b in parallel Dense A: Gaussian elimination with partial pivoting (LU) Same flavor as matrix * matrix, but more complicated Sparse A: Gaussian elimination Cholesky, LU, etc. Graph algorithms Sparse A:

589 views • 25 slides
Computing the rank of big sparse matrices modulo p using gaussian elimination Charles Bouillaguet

Computing the rank of big sparse matrices modulo p using gaussian elimination Charles Bouillaguet

Computing the rank of big sparse matrices modulo p using gaussian elimination Charles Bouillaguet 1 Claire Delaplace 2 12 CRIStAL, Universit de Lille 2 IRISA, Universit de Rennes 1 JNCF, 16 janvier 2017 Claire Delaplace Sparse elimination Mod

960 views • 54 slides
Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence Overview Motivation Sparse Gaussian Processes Stochastic Variational Inference Examples Overview Motivation Sparse Gaussian Processes

533 views • 36 slides
Switched behaviors with impulses A unifying framework Stephan Trenn and Jan C. Willems

Switched behaviors with impulses A unifying framework Stephan Trenn and Jan C. Willems

Switched behaviors with impulses A unifying framework Stephan Trenn and Jan C. Willems Technomathematics group, University of Kaiserslautern, Germany Department of Electrical Engineering, K.U. Leuven, Belgium 51st IEEE

461 views • 19 slides
Inferring Sparse Gaussian Graphical Models for Biological Network Christophe Ambroise Camille

Inferring Sparse Gaussian Graphical Models for Biological Network Christophe Ambroise Camille

Inferring Sparse Gaussian Graphical Models for Biological Network Christophe Ambroise Camille Charbonnier, Julien Chiquet, Gilles Grasseau, Catherine Matias, Yves Grandvalet Laboratoire Statistique et G enome, e d UMR CNRS 8071 -

1.48k views • 110 slides
Policy Interdependence in a Unifying Framework for a Monetary and Fiscal Union Yannis M.

Policy Interdependence in a Unifying Framework for a Monetary and Fiscal Union Yannis M.

Policy Interdependence in a Unifying Framework for a Monetary and Fiscal Union Yannis M. Ioannides Department of Economics, Tufts University Fletcher School: Greece and the Euro April 12, 2019 Motivation National policy options Model, at

473 views • 30 slides
Generalized Matrix Factorizations as a Unifying Framework for Pattern Set Mining Complexity

Generalized Matrix Factorizations as a Unifying Framework for Pattern Set Mining Complexity

Generalized Matrix Factorizations as a Unifying Framework for Pattern Set Mining Complexity Beyond Blocks Pauli Miettinen 10 September 2015 Community detection A B C ( ) 1 1 1 0 1 A 2 1 1 1 0 1 1 3 A B C B ( ) ( ) 2

301 views • 18 slides
Prior Knowledge and Sparse Methods for Convolved Multiple Outputs Gaussian Processes Mauricio A.

Prior Knowledge and Sparse Methods for Convolved Multiple Outputs Gaussian Processes Mauricio A.

Prior Knowledge and Sparse Methods for Convolved Multiple Outputs Gaussian Processes Mauricio A. lvarez Joint work with Neil D. Lawrence, David Luengo and Michalis K. Titsias School of Computer Science University of Manchester (University of

826 views • 78 slides
Sparse Gaussian Processes with Spherical Harmonic Features Vincent Dutordoir 1 , Nicolas Durrande 1

Sparse Gaussian Processes with Spherical Harmonic Features Vincent Dutordoir 1 , Nicolas Durrande 1

Sparse Gaussian Processes with Spherical Harmonic Features Vincent Dutordoir 1 , Nicolas Durrande 1 and James Hensman 2 1 PROWLER.io, 2 Amazon (Work completed while JH was at PROWLER.io) International Conference of Machine Learning 2020

666 views • 23 slides
Lecture 13 Gaussian Process Models - Part 2 3/06/2018 1 EDA and GPs 2 Variogram When fitting

Lecture 13 Gaussian Process Models - Part 2 3/06/2018 1 EDA and GPs 2 Variogram When fitting

Lecture 13 Gaussian Process Models - Part 2 3/06/2018 1 EDA and GPs 2 Variogram When fitting a Gaussian process model, it is often difficult to fit the covariance parameters (hard to identify). Today we will discuss some EDA approaches for

1.15k views • 49 slides
Bayesian Neural Networks from a Gaussian Process Perspective Andrew Gordon Wilson

Bayesian Neural Networks from a Gaussian Process Perspective Andrew Gordon Wilson

Bayesian Neural Networks from a Gaussian Process Perspective Andrew Gordon Wilson https://cims.nyu.edu/~andrewgw Courant Institute of Mathematical Sciences Center for Data Science New York University Gaussian Process Summer School September

508 views • 47 slides
Outline 1. Motivation 2. Gaussian process introduction 3. Change surface model 4. Analysis of

Outline 1. Motivation 2. Gaussian process introduction 3. Change surface model 4. Analysis of

5/22/2016 Scalable Gaussian Processes for Characterizing Multidimensional Change Surfaces April 18, 2016 William Herlands Committee: Daniel Neill, Alex Smola, Wilbert Van Panhuis Chair: Dave Choi Outline 1. Motivation 2. Gaussian process

1k views • 21 slides
Sparse Representations The Sparse Representations (SRs) framework was inspired by studies of

Sparse Representations The Sparse Representations (SRs) framework was inspired by studies of

Manifold Based Sparse Representation for Robust Expression Recognition without Neutral Subtraction Raymond Ptucha, Grigorios Tsagkatakis, Andreas Savakis Department of Computer Engineering, Rochester Institute of Technology, Rochester, NY Nov

411 views • 11 slides
CS475/CS675 Lecture 4: May 12, 2016 Sparse Gaussian Elimination, Graph Representation Reading:

CS475/CS675 Lecture 4: May 12, 2016 Sparse Gaussian Elimination, Graph Representation Reading:

CS475/CS675 Lecture 4: May 12, 2016 Sparse Gaussian Elimination, Graph Representation Reading: [Saad] Sect 3.13.2 CS475/CS675 (c) 2016 P. Poupart & J. Wan 1 5Point Stencil An easy way to denote 2D finite difference equations

450 views • 15 slides
Ruin problem for integrated stationary Gaussian process Kobelkov S. G. 1 Consider a random process

Ruin problem for integrated stationary Gaussian process Kobelkov S. G. 1 Consider a random process

Ruin problem for integrated stationary Gaussian process Kobelkov S. G. 1 Consider a random process t 0 X s ds ct , Y t = where X t , t 0 is a stationary real-valued zero-mean Gaussian process with continuous trajectories and twice

343 views • 5 slides
Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall

Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall

Implicit surface modelling Spline regularization as a Gaussian process GPIS for 2D curves GPIS for 3D surfaces Summary Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall Cambridge, UK Gaussian Processes

647 views • 40 slides
Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua

Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua

Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua University Joint work with Mohammad Emtiyaz Khan Jun Zhu Gaussian Process mean function covariance function / kernel inducing points Posterior

483 views • 13 slides
A prior near-ignorance Gaussian Process model for nonparametric regression Francesca Mangili

A prior near-ignorance Gaussian Process model for nonparametric regression Francesca Mangili

Introduction Gaussian Process (GP) Imprecise GP (IGP) Constant mean IGP Application Conclusions A prior near-ignorance Gaussian Process model for nonparametric regression Francesca Mangili francesca@idsia.ch Istituto Dalle Molle di

309 views • 15 slides

More recommend