Bayesian leave-one-out cross-validation for large data Mns Magnusson - PowerPoint PPT Presentation

Sep 15, 2023 •354 likes •421 views

Bayesian leave-one-out cross-validation for large data Mns Magnusson (Aalto University) Michael Riis Andersen (Technical University of Denmark) Johan Jonasson (University of Gothenburg) Aki Vehtari (Aalto University) Motivation: Model

Bayesian leave-one-out cross-validation for large data Måns Magnusson (Aalto University) Michael Riis Andersen (Technical University of Denmark) Johan Jonasson (University of Gothenburg) Aki Vehtari (Aalto University)
Motivation: Model selection for large data • Bigger data sets and more complex models • We still need to evaluate and compare models • elpd M quantifies how model M generalizes to unseen data ˜ y i � elpd M = log p M (˜ y i | y ) p t (˜ y i ) d ˜ y i , True data generating process Posterior predictive distribution Expected log predictive density 2 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019
Leave-one-out cross-validation • Basic idea: Hold out observation i and predict y i based on y − i • Estimate elpd M using leave-one-out cross-validation (loo) � � n � n elpd loo = 1 log p M ( y i | y − i ) = 1 log p M ( y i | θ ) p M ( θ | y − i ) dθ n n i =1 i =1 • Desirable properties + almost unbiased for large n + straight-forward handling of hierarchical structures • Two major problems - Need to fit the model n times - Need to evaluate predictive densities n times 3 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019
Our contributions: Method � n elpd loo = 1 log p M ( y i | y − i ) n i =1 • We propose a fast approximation for elpd loo 1 Approximate full data posterior q M ( θ | y ) using Variational Bayes/Laplace 2 Compute p M ( y i | y − i ) using importance sampling with q M as proposal 3 Subsample the sum over n using the Hansen-Hurwitz estimator • Solves both problems with leave-one-out CV 1 Only need to fit the model once on the full data set 2 Predictive distributions p M ( y i | y − i ) are only needed for a small subset 4 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019
Our contributions: Results • Theoretical results (under regularity conditions) � p elpd loo → elpd loo for n → ∞ • Extensive empirical results 1 Variational Bayes, Laplace approx., MCMC 2 Bayesian linear regression 3 Hierarchical models • For more details, come see us at poster #231 • Thank you for listening! 5 DTU Compute Bayesian leave-one-out cross-validation for large data 7.6.2019

Recommend

Introduction to Data Science: Classifier n 1 n 1 k k Suppose you want to compare two

Classier evaluation Classier evaluation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Leave-one-out Cross-Validation Classier evaluation Classier evaluation Leave-one-out Cross-Validation Resampled validation set

984 views • 51 slides

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides

Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications

Higher Order Analysis of Bayesian Cross Validation in Regular Asymptotic Theory Information Geometry and Its Applications IV In honor of Prof. Amaris 80 th birthday June 12-17, 2016, Liblice, Czech Republic Sumio Watanabe Tokyo Institute of

486 views • 30 slides

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views • 58 slides

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J.

15-388/688 - Practical Data Science: Nonlinear modeling, cross-validation, and regularization J. Zico Kolter Carnegie Mellon University Fall 2019 1 Outline Example: return to peak demand prediction Overfitting, generalization, and cross

903 views • 67 slides

in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101

Accelerating Cross-Validation in Spark Using GPU Minsik Cho, Rajesh Bordawekar IBM TJW Research 1 Cross-Validation 101 [Wikipedia] Popular Model Validation Technique to avoid overfitting, for better generalization useful when not

430 views • 20 slides

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation

Importance-Weighted Cross- Importance-Weighted Cross- Validation for Covariate Shift Validation for Covariate Shift (1) (2) Masashi Sugiyama , Benjamin Blankertz , (2,3) (2) Matthias Krauledat , Guido Dornhege , (3,2) Klaus-Robert

623 views • 26 slides

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013

Stratified Cross-Validation in Multi-Label Classification Using Genetic Algorithms 7-8/02/2013 TIN2010-20900-C04 Albacete Index Introduction Multilabel Classification Cross-Validation and Stratified Cross-Validation Methods and

682 views • 43 slides

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method

Progress to Date in A3: Method Transfer, Partial Validation and Cross validation A3: Method Transfer, partial and cross validation Team members: In scope Life cycle of a method after first full validation or relation Team

252 views • 10 slides

5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1

5: Overtraining and Cross-validation Machine Learning and Real-world Data Helen Yannakoudakis 1 Computer Laboratory University of Cambridge Lent 2018 1 Based on slides created by Simone Teufel Last session: smoothing and significance testing

1.56k views • 15 slides

5: Overtraining and Cross-validation Machine Learning and Real-world Data Simone Teufel and Ann

5: Overtraining and Cross-validation Machine Learning and Real-world Data Simone Teufel and Ann Copestake Computer Laboratory University of Cambridge Lent 2017 Last session: smoothing and significance testing You looked at various possible

208 views • 16 slides

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high

From Transfac to HOCOMOCO: using cross-validation and human curation to take most from the high throughput data compiling a complete collection of transcription factor binding motifs Vsevolod J. Makeev Vavilov Institute of General Genetics,

721 views • 37 slides

Cross-validation and the Bootstrap In the section we discuss two resampling methods:

Cross-validation and the Bootstrap In the section we discuss two resampling methods: cross-validation and the bootstrap. These methods refit a model of interest to samples formed from the training set, in order to obtain additional

688 views • 44 slides

5: Overtraining and Cross-validation Machine Learning and Real-world Data Paula Buttery (based

5: Overtraining and Cross-validation Machine Learning and Real-world Data Paula Buttery (based on slides created by Simone Teufel) Computer Laboratory University of Cambridge Lent 2019 Last session: smoothing and significance testing You

580 views • 15 slides

STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12

Outline Last Time Cross-Validation STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12 April 2016 Outline Last Time Cross-Validation Outline Last Time Cross-Validation Outline Last Time

381 views • 25 slides

Criticality experiments and benchmarks for for validation of cross validation of cross sections:

Criticality experiments and benchmarks Criticality experiments and benchmarks for for validation of cross validation of cross sections: the neptunium sections: the neptunium case case L.S.Leong, L.Tassan-Got, L.Audouin, C. Paradela,

464 views • 21 slides

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic*

CSC411 Tutorial #3 Cross-Validation and Decision Trees February 3, 2016 Boris Ivanovic* csc411ta@cs.toronto.edu *Based on the tutorial given by Erin Grant, Ziyu Zhang, and Ali Punjani in previous years. Outline for Today Cross-Validation

355 views • 24 slides

Classification K-nearest neighbor classification D istance functions Choice of k Choice of k

Classification K-nearest neighbor classification D istance functions Choice of k Choice of k Leave-one-out cross validation K-fold cross validation Classification Error = Average classification error on K folds Linear Classification Linear

395 views • 22 slides

Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Review Multiple Regression with Interaction Terms Model Selection:

522 views • 33 slides

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using given data What makes a model Features Hyper-parameters that control the hypothesis space Example: depth of a decision tree, neural

328 views • 12 slides

Model validation through "Posterior predictive checking" and "Leave-one-out"

Workshop of the Bayes WG / IBS-DR G. Nehmiz Mainz, 2006-12-01 M. Knen-Bergmann Model validation through "Posterior predictive checking" and "Leave-one-out" 1 Overview The posterior predictive distribution from a fitted

472 views • 30 slides

Introduction Types of Charts Data Tables Summarizing Data Cross-Tabulation

Introduction Types of Charts Data Tables Summarizing Data Cross-Tabulation Data visualization - the process of displaying data (often in large quantities) in a meaningful fashion to provide insights that will support better

1.07k views • 80 slides