Missing Data Imputation using Optimal Transport Boris Muzellec Julie - PowerPoint PPT Presentation

Dec 12, 2023 •366 likes •569 views

Missing Data Imputation using Optimal Transport Boris Muzellec Julie Josse Claire Boyer Marco Cuturi <latexit

Missing Data Imputation using Optimal Transport Boris Muzellec Julie Josse Claire Boyer Marco Cuturi
<latexit sha1_base64="xfKy83h87erRWERr/xzJ2E0Lv+E=">AB7HicbVDLTgJBEOzF+IL9ehlIjHxRHaNiR6JXjxiIo8ENmR26IUJM7ObmVkiIfyE8YZe/R3P/o0D7kHBOlV3Vae7K0oFN9b3v7zCxubW9k5xt7S3f3B4VD4+aZok0wbLBGJbkfUoOAKG5Zbge1UI5WRwFY0ul/orTFqwxP1ZCcphpIOFI85o9a12l0u3RY0vXLFr/pLkHUS5KQCOeq98me3n7BMorJMUGM6gZ/acEq15UzgrNTNDKaUjegAO/0xT42iEk04fV6ePCMXcaKJHSJZ1r/tUyqNmcjIeS1Q7OqLZr/aZ3MxrfhlKs0s6iYszgtzgSxCVl8TvpcI7Ni4ghlmrtDCRtSTZl1+ZRcAsHqv+ukeVUN/GrweF2p3eVZFOEMzuESAriBGjxAHRrAQMALzOHNU96rN/fef6wFL585hT/wPr4BZIGPEQ=</latexit> <latexit sha1_base64="xfKy83h87erRWERr/xzJ2E0Lv+E=">AB7HicbVDLTgJBEOzF+IL9ehlIjHxRHaNiR6JXjxiIo8ENmR26IUJM7ObmVkiIfyE8YZe/R3P/o0D7kHBOlV3Vae7K0oFN9b3v7zCxubW9k5xt7S3f3B4VD4+aZok0wbLBGJbkfUoOAKG5Zbge1UI5WRwFY0ul/orTFqwxP1ZCcphpIOFI85o9a12l0u3RY0vXLFr/pLkHUS5KQCOeq98me3n7BMorJMUGM6gZ/acEq15UzgrNTNDKaUjegAO/0xT42iEk04fV6ePCMXcaKJHSJZ1r/tUyqNmcjIeS1Q7OqLZr/aZ3MxrfhlKs0s6iYszgtzgSxCVl8TvpcI7Ni4ghlmrtDCRtSTZl1+ZRcAsHqv+ukeVUN/GrweF2p3eVZFOEMzuESAriBGjxAHRrAQMALzOHNU96rN/fef6wFL585hT/wPr4BZIGPEQ=</latexit> <latexit sha1_base64="xfKy83h87erRWERr/xzJ2E0Lv+E=">AB7HicbVDLTgJBEOzF+IL9ehlIjHxRHaNiR6JXjxiIo8ENmR26IUJM7ObmVkiIfyE8YZe/R3P/o0D7kHBOlV3Vae7K0oFN9b3v7zCxubW9k5xt7S3f3B4VD4+aZok0wbLBGJbkfUoOAKG5Zbge1UI5WRwFY0ul/orTFqwxP1ZCcphpIOFI85o9a12l0u3RY0vXLFr/pLkHUS5KQCOeq98me3n7BMorJMUGM6gZ/acEq15UzgrNTNDKaUjegAO/0xT42iEk04fV6ePCMXcaKJHSJZ1r/tUyqNmcjIeS1Q7OqLZr/aZ3MxrfhlKs0s6iYszgtzgSxCVl8TvpcI7Ni4ghlmrtDCRtSTZl1+ZRcAsHqv+ukeVUN/GrweF2p3eVZFOEMzuESAriBGjxAHRrAQMALzOHNU96rN/fef6wFL585hT/wPr4BZIGPEQ=</latexit> The missing data issue • Big data is plagued with missing values • What to do? Option 1: Remove entries with missing values information loss, not sustainable ⇒ = Example with 25% missing rate: 2d 3d 6d 10d With 1% missing rate: 5d: 95% rows kept 300d: 5% rows kept Option 2: Impute with reasonable guesses
Outline 1. Missing data and Optimal Transport 2. Non-parametric imputation with OT 3. Fitting parametric imputation models with OT
How to impute? - Mean imputation - Regression (conditional expectation) Deforms joint and marginal distributions Preserves distributions • Using a conditional model: - With logistic, multinomial, Poisson regressions: R’s mice (Van Buuren, 2011) • Assuming a joint model: - EM + Gaussian distribution: Amelia (Honacker et al., 2011) - Low-rank models: Softimpute (Mazumder et al., 2010) - VAE and GAN: MIWAE (Mattei & Frellsen, 2019), GAIN (Yoon et al., 2018) - … This work: Preserves distributions Parametric assumption not necessary

Recommend

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data Simple methods for dealing with missing data Single and multiple imputation R example Missing data is a complex problem We must consider:

377 views • 22 slides

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1.

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1. Missing Data and Missing Data Mechanisms 2. Imputation 3. Missing Data and Multiple Imputation in Baseline KLoSA Data Missing Data and Multiple

634 views • 35 slides

Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES,

WHAT ARE MISSING DATA ? HOW TO TREAT MISSING DATA ? LONGITUDINAL DATA, CAUSALITY, & ETHICS Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES, Universit de Lausanne FORS SHP workshop June

1.2k views • 108 slides

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 ,

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 , Ihab F. Ilyas 1 Theodoros Rekatsinas 2 1 2 Problem Missing data is a persistent problem in many fields Sciences Data mining

322 views • 28 slides

Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing

DataCamp Dealing With Missing Data in R DEALING WITH MISSING DATA IN R Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing Data in R Lesson overview Using imputations to understand data structure

647 views • 53 slides

Handling missing data in Stata: Imputation and likelihood-based approaches Rose Medeiros

Introduction Multiple Imputation Full information maximum likelihood Conclusion Handling missing data in Stata: Imputation and likelihood-based approaches Rose Medeiros StataCorp LP 2016 Swiss Stata Users Group meeting Medeiros Handling

841 views • 33 slides

MixtComp software: Model-based clustering/imputation with mixed data, missing data and uncertain

Classifications(s): overview Mixture model solution Estimation Clustering with MixtComp Imputation with MixtComp Conclusion MixtComp software: Model-based clustering/imputation with mixed data, missing data and uncertain data

966 views • 58 slides

N 1 N 1 IV x i y i s i z i s i z i (2) 3. Imputation i 1

A Course in Applied Econometrics 1 . When Can Missing Data be Ignored ? Lecture 18 : Missing Data Linear model with IVs: y i x i u i , (1) Jeff Wooldridge IRP Lectures, UW Madison, August 2008 where x i is 1 K , instruments z

236 views • 10 slides

Recognition of Reverberant Speech by Missing Data Imputation and NMF Feature Enhancement Heikki

Recognition of Reverberant Speech by Missing Data Imputation and NMF Feature Enhancement Heikki Kallasjoki , Jort F . Gemmeke, Kalle J. Palomki, Amy V. Beeston, Guy J. Brown Department of Signal Processing and Acoustics Aalto University,

687 views • 36 slides

Reference based multiple imputation; for sensitivity analysis of clinical trials with missing

Reference based multiple imputation; for sensitivity analysis of clinical trials with missing data Suzie Cro MRC Clinical Trials Unit at UCL The London School of Hygiene and Tropical Medicine Outline Reference based multiple imputation;

586 views • 39 slides

Incremental Algorithms for Missing Data Imputation based on Recursive Partitioning Claudio

Incremental Algorithms for Missing Data Imputation based on Recursive Partitioning Claudio Conversano Department of Economics University of Cassino, via M. Mazzaroppi, I-03043 Cassino (FR) c.conversano@unicas.it, http//cds.unina.it/~conversa

810 views • 38 slides

Accurate Regression Parameters and Summary Statistics Estimation in Data with Censored Missing

Accurate Regression Parameters and Summary Statistics Estimation in Data with Censored Missing Values Yuliya Karpievitch Imputation And Censored Values Yuliya Karpievitch Missing Values and Imputation Some algorithms expect a

531 views • 16 slides

Continuous Imputation of Missing Values in Streams of Pattern-Determining Time Series Kevin

Continuous Imputation of Missing Values in Streams of Pattern-Determining Time Series Kevin Wellenzohn 1 ohlen 1 Michael H. B os 2 Johann Gamper 2 Hannes Mitterer 2 Anton Dign 1 Department of Computer Science University of Zurich 2 Faculty

356 views • 31 slides

Martingale Optimal Transport in Higher Hadrien De March Dimension Optimal transport

MOTDim d Martingale Optimal Transport in Higher Hadrien De March Dimension Optimal transport Formulation of the problems Optimal Hadrien De March transport in practice Martingale CMAP, Ecole Polytechnique optimal transport

687 views • 24 slides

Imputation of missing covariates: when standard methods may fail Nicole S. Erler 1 , 2 , Dimitris

Imputation of missing covariates: when standard methods may fail Nicole S. Erler 1 , 2 , Dimitris Rizopoulos 1 , Oscar H. Franco 2 , Emmanuel M.E.H. Lesaffre 1 , 3 1 Department of Biostatistics, Erasmus MC, Rotterdam, the Netherlands 2 Department

678 views • 38 slides

Optimal Transport for structured data with application on graphs Titouan Vayer Joint work with

Optimal Transport for structured data with application on graphs Titouan Vayer Joint work with Laetitia Chapel, Remi Flamary, Romain Tavenard and Nicolas Courty A novel distance between labeled graphs based on optimal transport Contributions:

357 views • 15 slides

Advanced Section #2: Optimal Transport AC 209B: Data Science 2 Javier Zazo Pavlos Protopapas

Advanced Section #2: Optimal Transport AC 209B: Data Science 2 Javier Zazo Pavlos Protopapas Lecture Outline Historical overview Definitions and formulations Metric properties about optimal transport Application I: Supervised learning with

556 views • 44 slides

Merci pour votre attention E QUILIBRE DE N ASH & TRANSPORT OPTIMAL LJK G RENOBLE 3-4

E QUILIBRE DE N ASH & TRANSPORT OPTIMAL Adrien Blanchet TSE (GREMAQ, Universit e Toulouse 1 Capitole) Modelisation with optimal transport (ANR TOMMI) In collaboration with P . Mossay & F. Santambrogio and G. Carlier E QUILIBRE DE

463 views • 22 slides

Overview Multiple Imputation for Multilevel Data Bayesian estimation for MLMs Univariate

Overview Multiple Imputation for Multilevel Data Bayesian estimation for MLMs Univariate multiple imputation Session 1 Craig K. Enders Brian T. Keller Joint model imputation University of California - Los Angeles Fully conditional

901 views • 45 slides

Research questions... Research questions... Could missing data method change the quality of the

Research questions... Research questions... Could missing data method change the quality of the results obtained from a Customer Satisfaction market study? Could standard or classical imputation methods be applied no matter the rate of non

284 views • 3 slides

How black-box use of imputation can cause bias Nicole Erler Erasmus Medical Center, Rotterdam

How black-box use of imputation can cause bias Nicole Erler Erasmus Medical Center, Rotterdam Nicole Erler, FGME 2019, Kiel 1 Handling Missing Values is Easy! Functions automatically exclude missing values: ## [...] ## Residual standard

1.03k views • 43 slides

MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets Pierre-Alexandre Mattei

MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets Pierre-Alexandre Mattei IT University of Copenhagen http://pamattei.github.io/ @pamattei ICML 2019 Joint work with Jes Frellsen (ITU Copenhagen) 1 How to handle missing

353 views • 13 slides

Model-based clustering with mixed/missing data using the new software MixtComp

The problem Conditional independent clustering Estimation Clustering with MixtComp Imputation with MixtComp Conclusion Model-based clustering with mixed/missing data using the new software MixtComp

322 views • 29 slides

Dealing with missing values part 2 Applied Multivariate Statistics Spring 2012 Overview

Dealing with missing values part 2 Applied Multivariate Statistics Spring 2012 Overview More on Single Imputation: Shortcomings Multiple Imputation: Accounting for uncertainty Appl. Multivariate Statistics - Spring 2012 2 Single

676 views • 28 slides