doubly stochastic inference for deep gaussian processes
play

Doubly Stochastic Inference for Deep Gaussian Processes Hugh - PowerPoint PPT Presentation

Sep 05, 2022 •982 likes •1.53k views

Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing Imperial College London 29/5/2017 Motivation DGPs promise much, but are difficult to train 2 Doubly Stochastic Inference for DGPs Hugh

  1. Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing Imperial College London 29/5/2017

  2. Motivation § DGPs promise much, but are difficult to train 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  3. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  4. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well § We seek a variational approach that works and scales 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  5. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well § We seek a variational approach that works and scales 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  6. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well § We seek a variational approach that works and scales Other recently proposed schemes [1, 2, 5] make additional approximations and require more machinery than VI 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  7. Talk outline 1. Summary : Model Inference Results 2. Details: Model Inference Results 3. Questions 3 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  8. Model We use the standard DGP model, with one addition: 4 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  9. Model We use the standard DGP model, with one addition: § We include a linear (identity) mean function for all the internal layers 4 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  10. Model We use the standard DGP model, with one addition: § We include a linear (identity) mean function for all the internal layers (1D example in [4]) 4 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  11. Inference § We use the model conditioned on the inducing points as a conditional variational posterior 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  12. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  13. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) § We use sampling to deal with the intractable expectation 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  14. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) § We use sampling to deal with the intractable expectation 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  15. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) § We use sampling to deal with the intractable expectation We never compute N ˆ N matrices (we make no additional simplifications to variational posterior) 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  16. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  17. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  18. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  19. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better § We can get 98.1% on mnist with only 100 inducing points 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  20. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better § We can get 98.1% on mnist with only 100 inducing points § We surpass all permutation invariant methods on rectangles-images (designed to test deep vs shallow architectures) 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  21. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better § We can get 98.1% on mnist with only 100 inducing points § We surpass all permutation invariant methods on rectangles-images (designed to test deep vs shallow architectures) § Identical model/inference hyperparameters for all our models 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  22. Details: The Model We use the standard DGP model, with a linear mean function for all the internal layers: § If dimensions agree use the identity, otherwise PCA 7 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  23. Details: The Model We use the standard DGP model, with a linear mean function for all the internal layers: § If dimensions agree use the identity, otherwise PCA § Sensible alternative: initialize latents to identity (but linear mean function works better) 7 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  24. Details: The Model We use the standard DGP model, with a linear mean function for all the internal layers: § If dimensions agree use the identity, otherwise PCA § Sensible alternative: initialize latents to identity (but linear mean function works better) § Not so sensible alternative: random. Doesn’t work well (posterior is (very) multimodal) 7 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  25. The DGP: Graphical Model X Z 0 f 1 u 1 ✏ h 1 Z 1 f 2 u 2 ✏ h 2 Z 2 f 3 u 3 y 8 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  26. The DGP: Density likelihood hkkkkkikkkkkj N p p y , t h l , f l , u l u L π p p y i | f L l “ 1 q “ i q ˆ i “ 1 L π p p h l | f l q p p f l | u l ; h l ´ 1 , Z l ´ 1 q p p u l ; Z l ´ 1 q l “ 1 loooooooooooooooooooooooooomoooooooooooooooooooooooooon DGP prior 9 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  27. Factorised Variational Posterior X Z 0 X N ( u 1 | m 1 , S 1 ) f 1 u 1 f 1 u 1 ✏ 2 ) h 1 Z 1 i N ( h 1 i | µ 1 i , σ 1 h 1 Q i N ( u 2 | m 2 , S 2 ) f 2 u 2 f 2 u 2 ✏ 2 ) h 2 Z 2 Q i N ( h 2 i | µ 2 i , σ 2 h 2 i N ( u 3 | m 3 , S 3 ) f 3 u 3 f 3 u 3 y y 10 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  28. Our Variational Posterior X Z 0 X N ( u 1 | m 1 , S 1 ) f 1 u 1 f 1 u 1 ✏ ✏ h 1 Z 1 h 1 N ( u 2 | m 2 , S 2 ) f 2 u 2 f 2 u 2 ✏ ✏ h 2 Z 2 h 2 N ( u 3 | m 3 , S 3 ) f 3 u 3 f 3 u 3 y f 3 11 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  29. Recap: ‘GPs for Big Data’ [3] q p f , u q “ p p f | u ; X , Z q N p u | m , S q 12 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  30. Recap: ‘GPs for Big Data’ [3] q p f , u q “ p p f | u ; X , Z q N p u | m , S q Marginalise u from the variational posterior: ª p p f | u ; X , Z q N p u | m , S q d u “ N p f | µ , Σ q “ : q p f | m , S ; X , Z q (1) 12 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  31. Recap: ‘GPs for Big Data’ [3] q p f , u q “ p p f | u ; X , Z q N p u | m , S q Marginalise u from the variational posterior: ª p p f | u ; X , Z q N p u | m , S q d u “ N p f | µ , Σ q “ : q p f | m , S ; X , Z q (1) Define the following mean and covariance functions: µ m , Z p x i q “ m p x i q ` α p x i q T p m ´ m p Z qq , Σ S , Z p x i , x j q “ k p x i , x j q ´ α p x i q T p k p Z , Z q ´ S q α p x j q . where α p x i q “ k p x i , Z q k p Z , Z q ´ 1 12 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

Recommend

Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables

Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables

Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables Q. Wang & Herke van Hoof Amsterdam Machine Learning Lab ICML 2020 1 / 69 Highlights in this Work 2 / 69 Highlights in this Work A

965 views • 69 slides
Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent Dutordoir, James Hensman, Marc P Deisenroth Problem setting Problem setting Bimodal density Problem setting Changes with input Problem setting

1.16k views • 43 slides
Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence Overview Motivation Sparse Gaussian Processes Stochastic Variational Inference Examples Overview Motivation Sparse Gaussian Processes

533 views • 36 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate Inference Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14 th , 2019 1

1.14k views • 44 slides
Stochastic (partial) differential equations and Gaussian processes Simo Srkk Aalto

Stochastic (partial) differential equations and Gaussian processes Simo Srkk Aalto

Stochastic (partial) differential equations and Gaussian processes Simo Srkk Aalto University, Finland Contents Basic ideas 1 Stochastic differential equations and Gaussian processes 2 Stochastic partial differential equations and

2.22k views • 79 slides
Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Implicit Posterior Variational Inference for Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low and Patrick Jaillet Department of Computer Science National University of Singapore Department of

511 views • 25 slides
Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics University of Sheffield GP summer school September 2017 Why use Gaussian processes? Why use Gaussian processes? A stochastic process is a collection

1.69k views • 152 slides
Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian Processes November 10, 2015 1 / 22 What are Gaussian processes? GPs let us do Bayesian inference on functions . Using GPs we can: Interpolate spatial

728 views • 54 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications,

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications,

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications, Challenges & Opportunities Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14

551 views • 38 slides
Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto

Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto

Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto University, Finland Why use S(P)DE solvers for GPs? The O ( n 3 ) computational complexity is always a challenge. Latent force models combine PDE/ODEs with

482 views • 12 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part IV) Theory & Code

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part IV) Theory & Code

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part IV) Theory & Code Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14 th , 2019 1 Outline 1

529 views • 11 slides
Stochastic Processes Will Perkins March 7, 2013 Stochastic Processes Q: What is a Stochastic

Stochastic Processes Will Perkins March 7, 2013 Stochastic Processes Q: What is a Stochastic

Stochastic Processes Will Perkins March 7, 2013 Stochastic Processes Q: What is a Stochastic Process? A: A collection of random variables defined on the same probability space and indexed by a time parameter. { Z t } t T where each Z

1.52k views • 17 slides
Assembling stochastic quasi-Newton algorithms using Gaussian processes Thomas Sch on, Uppsala

Assembling stochastic quasi-Newton algorithms using Gaussian processes Thomas Sch on, Uppsala

Assembling stochastic quasi-Newton algorithms using Gaussian processes Thomas Sch on, Uppsala University, Sweden. Joint work with Adrian Wills , University of Newcastle, Australia. LCCC Workshop on Learning and Adaptation for Sensorimotor

434 views • 39 slides
Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim

Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim

Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim 2 , Will Tebbutt 1 , J. Scott Hosking 3 , 4 , Arno Solin 5 , Richard E. Turner 1 , 6 1 University of Cambridge, 2 Invenia Labs, 3 British Antarctic

340 views • 23 slides
Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge with Thang Bui, Yingzhen Li, Jos e Miguel Hern

957 views • 69 slides
Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the Search for Dark Matter with Machine Learning April 10, 2019 Based on Published in ICLR 2018, htups://arxiv.org/abs/1711.00165 Open source

501 views • 33 slides
Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian posteriors Approximate methods Laplace approximation Variational bayes Expectation propagation Comparisons GP regression - recap so far Model the

1.41k views • 113 slides
Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes Pierre-Franois Lget, Stanford University - KIPAC Gaussian process (GP) interpolation: Gaussian processes are a non-parametric way to interpolate data

281 views • 7 slides
State Space Expectation Propagation Efficient Inference Schemes for Temporal Gaussian Processes

State Space Expectation Propagation Efficient Inference Schemes for Temporal Gaussian Processes

IJ Aalto University State Space Expectation Propagation Efficient Inference Schemes for Temporal Gaussian Processes William Wilkinson , Paul Chang , Michael Riis Andersen , Arno Solin Aalto University , Technical University of

927 views • 45 slides
CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian Processes Fancier Gaussian Processes The current DFF. ( de facto fanciness) Uses for: Regression Classification

754 views • 50 slides
Fractional Gaussian Noise, Fractional Gaussian Noise, Subdiffusion and Stochastic and Stochastic

Fractional Gaussian Noise, Fractional Gaussian Noise, Subdiffusion and Stochastic and Stochastic

Fractional Gaussian Noise, Fractional Gaussian Noise, Subdiffusion and Stochastic and Stochastic Subdiffusion Networks in Biophysics Networks in Biophysics Samuel Kou Department of Statistics Harvard University Single-Molecule Experiments

741 views • 46 slides
State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2 Alexander Grigorievskiy 2 , 3 1 Philips Research, 2 Aalto University, 3 Silo.AI ICML2018 July 13, 2018 Outline Gaussian Processes Temporal GPs as

285 views • 17 slides
Stochastic Processes, Kernel Regression, Infinite Mixture Models Gabriel Huang (TA for Simon

Stochastic Processes, Kernel Regression, Infinite Mixture Models Gabriel Huang (TA for Simon

Stochastic Processes, Kernel Regression, Infinite Mixture Models Gabriel Huang (TA for Simon Lacoste-Julien) IFT 6269 : Probabilistic Graphical Models - Fall 2018 Stochastic Process = Random Function 2 Today Motivate Gaussian and

1.13k views • 79 slides
Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil

Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil

Robust, Deep Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil Lawrence, Jeremy Forth, Guilherme Barreto Royal Society , 06 June 2016 Challenge: Learn patterns from sequences Recurrent Gaussian

622 views • 25 slides

More recommend