power expectation propagation for deep gaussian processes
play

Power Expectation Propagation for Deep Gaussian Processes Dr. - PowerPoint PPT Presentation

Feb 05, 2023 •253 likes •957 views

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge with Thang Bui, Yingzhen Li, Jos e Miguel Hern

  1. Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge with Thang Bui, Yingzhen Li, Jos´ e Miguel Hern´ andez Lobato, Daniel Hern´ andez Lobato, Josiah Jan 1 / 32

  2. Motivation: Gaussian Process regression outputs inputs 2 / 32

  3. Motivation: Gaussian Process regression ? outputs inputs 2 / 32

  4. Motivation: Gaussian Process regression ? outputs inputs 2 / 32

  5. Motivation: Gaussian Process regression inference & learning ? outputs inputs 2 / 32

  6. Motivation: Gaussian Process regression inference & learning intractabilities computational analytic ? outputs inputs 2 / 32

  7. EP pseudo-point approximation true posterior 3 / 32

  8. EP pseudo-point approximation true posterior 3 / 32

  9. EP pseudo-point approximation marginal posterior likelihood true posterior 3 / 32

  10. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 3 / 32

  11. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 3 / 32

  12. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 3 / 32

  13. EP pseudo-point approximation marginal posterior likelihood true posterior approximate posterior 3 / 32

  14. EP pseudo-point approximation marginal posterior likelihood approximate posterior true posterior input locations of outputs and covariance 'pseudo' data 'pseudo' data 3 / 32

  15. EP algorithm 4 / 32

  16. EP algorithm take out one 1. remove pseudo-observation likelihood cavity 4 / 32

  17. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood tilted 4 / 32

  18. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 4 / 32

  19. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family update 4. update pseudo-observation likelihood 4 / 32

  20. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 1. minimum: moments matched at pseudo-inputs 2. Gaussian regression: matches moments everywhere update 4. update pseudo-observation likelihood 4 / 32

  21. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 1. minimum: moments matched at pseudo-inputs 2. Gaussian regression: matches moments everywhere update 4. update pseudo-observation likelihood rank 1 4 / 32

  22. Fixed points of EP = FITC approximation 5 / 32

  23. Fixed points of EP = FITC approximation suppressed & 5 / 32

  24. Fixed points of EP = FITC approximation suppressed & 5 / 32

  25. Fixed points of EP = FITC approximation suppressed & 5 / 32

  26. Fixed points of EP = FITC approximation suppressed & 6 / 32

  27. Fixed points of EP = FITC approximation suppressed & 6 / 32

  28. Fixed points of EP = FITC approximation suppressed & 6 / 32

  29. Fixed points of EP = FITC approximation suppressed & 7 / 32

  30. Fixed points of EP = FITC approximation suppressed & 7 / 32

  31. Fixed points of EP = FITC approximation suppressed & 7 / 32

  32. Fixed points of EP = FITC approximation suppressed & 7 / 32

  33. Fixed points of EP = FITC approximation suppressed & 8 / 32

  34. Fixed points of EP = FITC approximation suppressed & = equivalent 8 / 32

  35. Fixed points of EP = FITC approximation suppressed & = Csato & Opper (2002) equivalent Qi, Abdel-Gawad & Minka (2010) 8 / 32

  36. Fixed points of EP = FITC approximation suppressed & = Csato & Opper (2002) equivalent Qi, Abdel-Gawad & Minka (2010) Interpretation resolves philosophical issues with FITC (increase M with N) FITC known to overfit => EP over-estimates marginal likelihood 8 / 32

  37. EP algorithm take out one 1. remove pseudo-observation likelihood cavity add in one 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 1. minimum: moments matched at pseudo-inputs 2. Gaussian regression: matches moments everywhere update 4. update pseudo-observation likelihood rank 1 9 / 32

  38. Power EP algorithm (as tractable as EP) take out fraction of 1. remove pseudo-observation likelihood cavity add in fraction of 2. include true observation likelihood KL between unnormalised tilted stochastic processes project onto 3. project approximating family 1. minimum: moments matched at pseudo-inputs 2. Gaussian regression: matches moments everywhere update 4. update pseudo-observation likelihood rank 1 10 / 32

  39. Power EP: a unifying framework FITC VFE Csato and Opper, 2002 Titsias, 2009 Snelson and Ghahramani, 2005 11 / 32

  40. Power EP: a unifying framework Approximate blocks of data: structured approximations 12 / 32

  41. Power EP: a unifying framework Approximate blocks of data: structured approximations 12 / 32

  42. Power EP: a unifying framework Approximate blocks of data: structured approximations PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009 12 / 32

  43. Power EP: a unifying framework Approximate blocks of data: structured approximations PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009 Place pseudo-data in different space: interdomain transformations (linear transform) 12 / 32

  44. Power EP: a unifying framework Approximate blocks of data: structured approximations PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009 Place pseudo-data in different space: interdomain transformations (linear transform) 12 / 32

  45. Power EP: a unifying framework Approximate blocks of data: structured approximations PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009 Place pseudo-data in different space: interdomain transformations (linear transform) pseudo-data in new space 12 / 32

  46. Power EP: a unifying framework Approximate blocks of data: structured approximations PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009 Place pseudo-data in different space: interdomain transformations (linear transform) Figueiras-Vidal & La ́ zaro-Gredilla 2009 pseudo-data in new space T obar et al. 2015 Matthews et al, 2016 12 / 32

  47. Power EP: a unifying framework GP Regression GP Classification ** ** [16*] inter-domain inter-domain [13] [17,13] [7,4*,6*] (PITC) [9,11,8*] [14*] structured structured [12*,15*] [10,5,6*] approx. approx. (FITC) VFE VFE PEP PEP EP EP [4] Quiñonero-Candela et al. 2005 [8] Titsias, 2009 [12] Naish-Guzman et al, 2007 [5] Snelson et al., 2005 [9] Csató, 2002 [13] Qi et al., 2010 [6] Snelson, 2006 [14] Hensman et al., 2015 [10] Csató et al., 2002 [7] Schwaighofer, 2002 [11] Seeger et al., 2003 [15] Hernández-Lobato et al., 2016 [16] Matthews et al., 2016 * = optimised pseudo-inputs [17] Figueiras-Vidal et al., 2009 ** = structured versions of VFE recover VFE 13 / 32

  48. How should I set the power parameter α ? 8 UCI regression datasets 6 UCI classification datasets 20 random splits 20 random splits M = 0 - 200 M = 10, 50, 100 hypers and inducing inputs optimised hypers and inducing inputs optimised 0.8 0.1 0.5 0.8 0.2 0.05 0.4 0 0 1 0.6 1 1 2 3 4 5 6 7 8 1 2 3 4 CD Error Average Rank SMSE Average Rank Error Average Rank indicates significant difference 0.4 0.2 0.8 1 0.1 1 0.5 0 0.6 0.05 0.8 0 1 2 3 4 1 2 3 4 5 6 7 8 CD SMLL Average Rank MLL Average Rank MLL Average Rank = 0.5 does well over all 14 / 32

  49. Deep Gaussian processes f l ∼ GP (0 , k ( ., . )) y n = g ( x n ) = f L ( f L − 1 ( · · · f 2 ( f 1 ( x n )))) + ǫ n x n f 1 h L − 1 , n := f L − 1 ( · · · f 1 ( x n )) , y n = f L ( h L − 1 , n ) + ǫ n h 1 , n Deep GPs a are f 2 multi-layer generalisation of Gaussian processes, equivalent to deep neural networks with infinitely h 2 , n wide hidden layers f 3 Questions: How to perform inference and learning tractably? y n How Deep GPs compare to alternative, e.g. Bayesian neural networks? N a Damianou and Lawrence (2013) [unsupervised learning] 15 / 32

  50. Pros and cons of Deep GPs Why deep GPs? Because x n deep and nonparametric, f 1 discover useful input warping or dimensionality compression/expansion, i.e. automatic, h 1 , n nonparametric Bayesian kernel design, f 2 give a non-Gaussian functional mapping g , h 2 , n f 3 y n N 16 / 32

Recommend

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

884 views • 71 slides
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

715 views • 68 slides
Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian posteriors Approximate methods Laplace approximation Variational bayes Expectation propagation Comparisons GP regression - recap so far Model the

1.41k views • 113 slides
State Space Expectation Propagation Efficient Inference Schemes for Temporal Gaussian Processes

State Space Expectation Propagation Efficient Inference Schemes for Temporal Gaussian Processes

IJ Aalto University State Space Expectation Propagation Efficient Inference Schemes for Temporal Gaussian Processes William Wilkinson , Paul Chang , Michael Riis Andersen , Arno Solin Aalto University , Technical University of

927 views • 45 slides
Scalable Multi-Class Gaussian Process Classification using Expectation Propagation Carlos

Scalable Multi-Class Gaussian Process Classification using Expectation Propagation Carlos

Scalable Multi-Class Gaussian Process Classification using Expectation Propagation Carlos Villacampa-Calvo and Daniel Hern andezLobato Computer Science Department Universidad Aut onoma de Madrid http://dhnzl.org ,

963 views • 69 slides
Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the Search for Dark Matter with Machine Learning April 10, 2019 Based on Published in ICLR 2018, htups://arxiv.org/abs/1711.00165 Open source

501 views • 33 slides
Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes Pierre-Franois Lget, Stanford University - KIPAC Gaussian process (GP) interpolation: Gaussian processes are a non-parametric way to interpolate data

281 views • 7 slides
CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian Processes Fancier Gaussian Processes The current DFF. ( de facto fanciness) Uses for: Regression Classification

754 views • 50 slides
State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2 Alexander Grigorievskiy 2 , 3 1 Philips Research, 2 Aalto University, 3 Silo.AI ICML2018 July 13, 2018 Outline Gaussian Processes Temporal GPs as

285 views • 17 slides
Power Variation and Gaussian Processes with Stationary Increments Jos M. Corcuera Faculty of

Power Variation and Gaussian Processes with Stationary Increments Jos M. Corcuera Faculty of

Power Variation and Gaussian Processes with Stationary Increments Jos M. Corcuera Faculty of Mathematics University of Barcelona Stochastics in Turbulence and Finance, Sandbjerg, 31 January 2008 fblanc1 Jos M. Corcuera (University of

583 views • 55 slides
Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Implicit Posterior Variational Inference for Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low and Patrick Jaillet Department of Computer Science National University of Singapore Department of

511 views • 25 slides
Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil

Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil

Robust, Deep Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil Lawrence, Jeremy Forth, Guilherme Barreto Royal Society , 06 June 2016 Challenge: Learn patterns from sequences Recurrent Gaussian

622 views • 25 slides
Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics University of Sheffield GP summer school September 2017 Why use Gaussian processes? Why use Gaussian processes? A stochastic process is a collection

1.69k views • 152 slides
Gaussian Process Regression with Perry Groot , Peter Lucas Censored Data Using Expectation

Gaussian Process Regression with Perry Groot , Peter Lucas Censored Data Using Expectation

GP Regression with Censored Data using EP Gaussian Process Regression with Perry Groot , Peter Lucas Censored Data Using Expectation Introduction Propagation Problem Setting Bayesian Framework Perry Groot , Peter Lucas Prior - GPs

554 views • 17 slides
Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian Processes November 10, 2015 1 / 22 What are Gaussian processes? GPs let us do Bayesian inference on functions . Using GPs we can: Interpolate spatial

728 views • 54 slides
An empirical study of Gaussian belief propagation and application in the detection of F-formations

An empirical study of Gaussian belief propagation and application in the detection of F-formations

Regularized Gaussian belief propagation Empirical Study F-formation detection Concluding Remarks An empirical study of Gaussian belief propagation and application in the detection of F-formations Francois Kamper Stellenbosch University

578 views • 20 slides
Expectation Maximization Greg Mori - CMPT 419/726 Bishop PRML Ch. 9 K-Means Gaussian Mixture

Expectation Maximization Greg Mori - CMPT 419/726 Bishop PRML Ch. 9 K-Means Gaussian Mixture

K-Means Gaussian Mixture Models Expectation-Maximization Expectation Maximization Greg Mori - CMPT 419/726 Bishop PRML Ch. 9 K-Means Gaussian Mixture Models Expectation-Maximization Learning Parameters to Probability Distributions We

1.37k views • 91 slides
Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing

Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing

Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing Imperial College London 29/5/2017 Motivation DGPs promise much, but are difficult to train 2 Doubly Stochastic Inference for DGPs Hugh

1.53k views • 53 slides
CSci 8980: Advanced Topics in Graphical Models Expectation Propagation Instructor: Arindam

CSci 8980: Advanced Topics in Graphical Models Expectation Propagation Instructor: Arindam

Posterior Estimation Assumed Density Filtering Expectation Propagation Experiments CSci 8980: Advanced Topics in Graphical Models Expectation Propagation Instructor: Arindam Banerjee October 26, 2007 Posterior Estimation Assumed Density

1.02k views • 63 slides
Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent Dutordoir, James Hensman, Marc P Deisenroth Problem setting Problem setting Bimodal density Problem setting Changes with input Problem setting

1.16k views • 43 slides
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee

CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee

CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian Processes Outline Parametric

572 views • 40 slides
Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression Predictions using noisy observations The case of a single test input: Gaussian Process Regression Computational and numerical issues It is

444 views • 29 slides
Probabilistic & Unsupervised Learning Expectation Propagation Maneesh Sahani

Probabilistic & Unsupervised Learning Expectation Propagation Maneesh Sahani

Probabilistic & Unsupervised Learning Expectation Propagation Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1, Autumn 2018

1.12k views • 109 slides
Probabilistic & Unsupervised Learning Expectation Propagation Maneesh Sahani

Probabilistic & Unsupervised Learning Expectation Propagation Maneesh Sahani

Probabilistic & Unsupervised Learning Expectation Propagation Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1, Autumn 2019

1.44k views • 132 slides

More recommend