deep gaussian processes with importance weighted
play

Deep Gaussian Processes with Importance-Weighted Variational - PowerPoint PPT Presentation

Apr 10, 2024 •715 likes •1.16k views

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent Dutordoir, James Hensman, Marc P Deisenroth Problem setting Problem setting Bimodal density Problem setting Changes with input Problem setting

  1. Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent Dutordoir, James Hensman, Marc P Deisenroth

  2. Problem setting

  3. Problem setting Bimodal density

  4. Problem setting Changes with input

  5. Problem setting Skewness

  6. Problem setting Skewness • Bus arrival times

  7. Problem setting Skewness • Bus arrival times • Confounding variables

  8. A possible approach f φ N y n x n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  9. A possible approach test samples training data f φ N y n x n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  10. A possible approach test samples Neural network training data f φ N y n x n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  11. A possible approach test samples Neural network training data f φ Latent variable N (per point) y n x n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  12. A possible approach test samples Neural network training data f φ Latent variable N (per point) y n x n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Concatenation with inputs

  13. A possible approach f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  14. A possible approach f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  15. A possible approach f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  16. A possible approach f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1)

  17. A possible approach f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Unreliable extrapolation

  18. A possible approach Overfitting f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Unreliable extrapolation

  19. A possible approach Overfitting Deterministic function f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Unreliable extrapolation

  20. A possible approach Overfitting Deterministic function f φ N x n y n w n y n = N ( f φ ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Unreliable extrapolation Small number of examples per input x n

  21. Another possible approach ∞ f N y n x n w n y n = N ( f ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ, k )

  22. Another possible approach Non-parametric prior ∞ f N y n x n w n y n = N ( f ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ, k )

  23. Another possible approach Non-parametric prior ∞ f N y n x n w n y n = N ( f ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Better extrapolation f ∼ GP ( µ, k )

  24. Another possible approach Non-parametric prior ∞ f N y n x n w n y n = N ( f ([ x n , w n ]) , σ 2 ) w n ∼ N (0 , 1) Better extrapolation f ∼ GP ( µ, k ) Underfitting

  25. Our model ∞ ∞ g f N x n y n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ 1 , k 1 ) g ∼ GP ( µ 2 , k 2 )

  26. Our model ∞ ∞ g f N x n y n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ 1 , k 1 ) g ∼ GP ( µ 2 , k 2 )

  27. Our model ∞ ∞ g f N x n y n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ 1 , k 1 ) g ∼ GP ( µ 2 , k 2 )

  28. Our model Extrapolating gracefully ∞ ∞ g f N x n y n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ 1 , k 1 ) g ∼ GP ( µ 2 , k 2 )

  29. Our model Extrapolating gracefully ∞ ∞ g f N x n y n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ 1 , k 1 ) Better data fit g ∼ GP ( µ 2 , k 2 )

  30. Contributions

  31. Contributions • New architecture - latent variables by concatenation, not addition

  32. Contributions • New architecture - latent variables by concatenation, not addition • Importance-weighted variational inference, exploiting analytic results

  33. Contributions • New architecture - latent variables by concatenation, not addition • Importance-weighted variational inference, exploiting analytic results • Provide an extensive empirical comparison with all 41 UCI regression datasets

  34. A few details ∞ ∞ g f N y n x n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) w n ∼ N (0 , 1) f ∼ GP ( µ 1 , k 1 ) g ∼ GP ( µ 2 , k 2 )

  35. A few details ∞ ∞ g f N y n x n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) Importance weighting w n ∼ N (0 , 1) (Gaussian proposal) f ∼ GP ( µ 1 , k 1 ) g ∼ GP ( µ 2 , k 2 )

  36. A few details ∞ ∞ g f N y n x n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) Importance weighting w n ∼ N (0 , 1) (Gaussian proposal) Variational inference f ∼ GP ( µ 1 , k 1 ) (sparse GP posterior) g ∼ GP ( µ 2 , k 2 )

  37. A few details ∞ ∞ g f N y n x n w n y n = N ( f ( g ([ x n , w n ])) , σ 2 ) Importance weighting w n ∼ N (0 , 1) (Gaussian proposal) Variational inference f ∼ GP ( µ 1 , k 1 ) (sparse GP posterior) g ∼ GP ( µ 2 , k 2 ) Our approach exploits analytic results, leading to a tighter bound

  38. Results

  39. Results • Latent variables in the DGP are highly beneficial

  40. Results • Latent variables in the DGP are highly beneficial • Sometimes depth is enough. Sometimes latent variables are enough. Some datasets need both .

  41. Results • Latent variables in the DGP are highly beneficial • Sometimes depth is enough. Sometimes latent variables are enough. Some datasets need both . • Importance-weighted VI outperforms VI

  42. Results • Latent variables in the DGP are highly beneficial • Sometimes depth is enough. Sometimes latent variables are enough. Some datasets need both . • Importance-weighted VI outperforms VI

  43. Thanks for listening Poster #218 • New architecture • Importance-weighted • 41 datasets

Recommend

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge with Thang Bui, Yingzhen Li, Jos e Miguel Hern

957 views • 69 slides
On Casting Importance Weighted Autoencoder to an EM Algorithm to Learn Deep Generative Models

On Casting Importance Weighted Autoencoder to an EM Algorithm to Learn Deep Generative Models

On Casting Importance Weighted Autoencoder to an EM Algorithm to Learn Deep Generative Models D.Kim 1 and J.Hwang 2 and Y.Kim 1 Speaker : Dongha Kim 1 Department of Statistics, Seoul National University, South Korea 2 SK Telecom, South Korea

671 views • 31 slides
Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the Search for Dark Matter with Machine Learning April 10, 2019 Based on Published in ICLR 2018, htups://arxiv.org/abs/1711.00165 Open source

501 views • 33 slides
Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian posteriors Approximate methods Laplace approximation Variational bayes Expectation propagation Comparisons GP regression - recap so far Model the

1.41k views • 113 slides
Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes Pierre-Franois Lget, Stanford University - KIPAC Gaussian process (GP) interpolation: Gaussian processes are a non-parametric way to interpolate data

281 views • 7 slides
Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation for Covariate Shift (1) (2) Masashi Sugiyama , Benjamin Blankertz , (2,3) (2) Matthias Krauledat , Guido Dornhege , (3,2) Klaus-Robert

623 views • 26 slides
CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian

CMPUT 466 Introduction to Gaussian Processes Dan Lizotte The Plan Introduction to Gaussian Processes Fancier Gaussian Processes The current DFF. ( de facto fanciness) Uses for: Regression Classification

754 views • 50 slides
State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2

State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2 Alexander Grigorievskiy 2 , 3 1 Philips Research, 2 Aalto University, 3 Silo.AI ICML2018 July 13, 2018 Outline Gaussian Processes Temporal GPs as

285 views • 17 slides
Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Implicit Posterior Variational Inference for Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low and Patrick Jaillet Department of Computer Science National University of Singapore Department of

511 views • 25 slides
Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil

Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil

Robust, Deep Recurrent Gaussian processes Andreas Damianou with C esar Lincoln Mattos, Zhenwen Dai, Neil Lawrence, Jeremy Forth, Guilherme Barreto Royal Society , 06 June 2016 Challenge: Learn patterns from sequences Recurrent Gaussian

622 views • 25 slides
Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics University of Sheffield GP summer school September 2017 Why use Gaussian processes? Why use Gaussian processes? A stochastic process is a collection

1.69k views • 152 slides
Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian Processes November 10, 2015 1 / 22 What are Gaussian processes? GPs let us do Bayesian inference on functions . Using GPs we can: Interpolate spatial

728 views • 54 slides
Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing

Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing

Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing Imperial College London 29/5/2017 Motivation DGPs promise much, but are difficult to train 2 Doubly Stochastic Inference for DGPs Hugh

1.53k views • 53 slides
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee

CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee

CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian Processes Outline Parametric

572 views • 40 slides
Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression Predictions using noisy observations The case of a single test input: Gaussian Process Regression Computational and numerical issues It is

444 views • 29 slides
Hierarchical Importance Weighted Autoencoders Chin-Wei Huang Kris Sankaran Eeshan Dhekane

Hierarchical Importance Weighted Autoencoders Chin-Wei Huang Kris Sankaran Eeshan Dhekane

Hierarchical Importance Weighted Autoencoders Chin-Wei Huang Kris Sankaran Eeshan Dhekane Alexandre Lacoste Aaron Courville p(z|x) z~q(z) Importance weighted autoencoders (IWAE) larger K tighter bound reduce variance Variance reduction

289 views • 10 slides
My research over Bayesian Optimization and Gaussian Processes Eduardo C. GarridoMerch an

My research over Bayesian Optimization and Gaussian Processes Eduardo C. GarridoMerch an

My research over Bayesian Optimization and Gaussian Processes Eduardo C. GarridoMerch an PhD student. Teacher Assistant. Universidad Aut onoma de Madrid 1/13 Index Introduction: Bayesian Optimization and Gaussian Processes. 2/13

923 views • 70 slides
Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence Overview Motivation Sparse Gaussian Processes Stochastic Variational Inference Examples Overview Motivation Sparse Gaussian Processes

533 views • 36 slides
Practical Challenges of Gaussian Processes Marc Deisenroth Statistical Machine Learning Group

Practical Challenges of Gaussian Processes Marc Deisenroth Statistical Machine Learning Group

Practical Challenges of Gaussian Processes Marc Deisenroth Statistical Machine Learning Group Department of Computing Imperial College London New Directions for Learning with Kernels and Gaussian Processes, Dagstuhl November 29, 2016 Data

264 views • 5 slides
Scalable Gaussian processes with a twist of Probabilistic Numerics Kurt Cutajar EURECOM, Sophia

Scalable Gaussian processes with a twist of Probabilistic Numerics Kurt Cutajar EURECOM, Sophia

Scalable Gaussian processes with a twist of Probabilistic Numerics Kurt Cutajar EURECOM, Sophia Antipolis, France Data Science Meetup - October 30 th 2017 Agenda Kernel Methods Scalable Gaussian Processes (using Preconditioning)

481 views • 46 slides
Introduction to Gaussian Processes Stephen Keeley and Jonathan Pillow Princeton Neuroscience

Introduction to Gaussian Processes Stephen Keeley and Jonathan Pillow Princeton Neuroscience

Introduction to Gaussian Processes Stephen Keeley and Jonathan Pillow Princeton Neuroscience Institute Princeton University skeeley@princeton.edu March 28, 2018 Gaussian Processes (GPs) are a flexible and general way to parameterize functions

174 views • 5 slides
Gaussian Processes Covariance Functions and Classification Carl Edward Rasmussen Max Planck

Gaussian Processes Covariance Functions and Classification Carl Edward Rasmussen Max Planck

Gaussian Processes Covariance Functions and Classification Carl Edward Rasmussen Max Planck Institute for Biological Cybernetics T ubingen, Germany Gaussian Processes in Practice, Bletchley Park, July 12th, 2006 Carl Edward Rasmussen

696 views • 30 slides
CSC 411 Lecture 20: Gaussian Processes Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 20: Gaussian Processes Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 20: Gaussian Processes Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 20-Gaussian Processes 1 / 24 Overview Last lecture: Bayesian linear regression, a parametric model This

835 views • 24 slides
Gaussian Processes to Speed up Hamiltonian Monte Carlo Matthieu L Murray, Iain

Gaussian Processes to Speed up Hamiltonian Monte Carlo Matthieu L Murray, Iain

Gaussian Processes to Speed up Hamiltonian Monte Carlo Matthieu L Murray, Iain http://videolectures.net/mlss09uk_murray_mcmc/ Rasmussen, Carl Edward. "Gaussian processes to speed up hybrid Monte Carlo for expensive Bayesian

357 views • 14 slides

More recommend