working with missing data
play

Working with Missing Data Steve Borgatti LINKS Center Workshop on - PDF document

Jun 20, 2023 •16 likes •116 views

2009 LINKS Center Workshop on Social Network Analysis Slide 1 Working with Missing Data Steve Borgatti LINKS Center Workshop on Social Network Analysis _____________________________________________________________________________________

  1. 2009 LINKS Center Workshop on Social Network Analysis Slide 1 Working with Missing Data Steve Borgatti LINKS Center Workshop on Social Network Analysis _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ __________ FRIDAY (c) 2009 LINKS Center ADVANCED Session

  2. 2009 LINKS Center Workshop on Social Network Analysis Slide 2 The problem: 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 Some respondents did • A A B B C G G L M P P P R S S T not participate in the - - - - - - - - - - - - - - - - survey, leaving blank 1 ACCIAIUOL 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 ALBIZZI 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 rows in the network 3 BARBADORI 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 data matrix 4 BISCHERI Important note: if you 5 CASTELLAN 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 • 6 GINORI 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 enter data as edgelist 7 GUADAGNI 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 or nodelist using dl file, 8 LAMBERTES 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 missing values are 9 MEDICI 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 1 automatically 10 PAZZI 11 PERUZZI 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 converted to zeros. 12 PUCCI 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 How would you convert – 13 RIDOLFI 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 them back? 14 SALVIATI 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 15 STROZZI 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 16 TORNABUON 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ __________ FRIDAY (c) 2009 LINKS Center ADVANCED Session

  3. 2009 LINKS Center Workshop on Social Network Analysis Slide 3 Size of the problem • Counting the number of missing values with Tools | Freq. – Select “matrices” 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 A A B B C G G L M P P P R S S T - - - - - - - - - - - - - - - - 1 ACCIAIUOL 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 ALBIZZI 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 3 BARBADORI 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 4 BISCHERI 5 CASTELLAN 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 6 GINORI 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 GUADAGNI 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 8 LAMBERTES 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 9 MEDICI 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 1 10 PAZZI 11 PERUZZI 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 12 PUCCI 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 RIDOLFI 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 14 SALVIATI 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 15 STROZZI 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 16 TORNABUON 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 Output: 1 ----- 0.000 0.725 1.000 0.150 blank 0.125 _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ __________ FRIDAY (c) 2009 LINKS Center ADVANCED Session

  4. 2009 LINKS Center Workshop on Social Network Analysis Slide 4 Standard Solutions • Convert missings to zeros (since you did NOT observe a tie) – Re ‐ run having converted missings to ones, to see how different the results could be • Convert missings to zeros and ones at random, using density of the matrix as guide • Impute the missing values using other information – Symmetricity – QAP regression _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ __________ FRIDAY (c) 2009 LINKS Center ADVANCED Session

  5. 2009 LINKS Center Workshop on Social Network Analysis Slide 5 One solution 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 Suppose the data are • A A B B C G G L M P P P R S S T largely symmetric, and - - - - - - - - - - - - - - - - the social relation is 1 ACCIAIUOL 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 logically symmetric 2 ALBIZZI 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 3 BARBADORI 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 Marriage to – 4 BISCHERI Saw movie with – 5 CASTELLAN 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 Then we can impute the • 6 GINORI 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 missing data from the 7 GUADAGNI 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 transpose of the matrix 8 LAMBERTES 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 9 MEDICI 1 1 1 0 0 0 0 0 0 0 0 0 1 1 0 1 i.e., assume that if A says – 10 PAZZI B is a friend, then if B had 11 PERUZZI 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 participated, s/he would 12 PUCCI 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 have said A was a a friend too 13 RIDOLFI 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 14 SALVIATI 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 So, fill in missing row with – 15 STROZZI 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 the corresponding column 16 TORNABUON 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ __________ FRIDAY (c) 2009 LINKS Center ADVANCED Session

  6. 2009 LINKS Center Workshop on Social Network Analysis Slide 6 Doing it … • In matrix algebra, you can do it with the REPLACENA command: – newdata = replacena(olddata transp(olddata)) • Syntax – > <newds> = replacena(<ds1> <ds2>) – Where ds1 is the dataset that contains missing values and ds2 is the dataset from which to draw the correct values _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ __________ FRIDAY (c) 2009 LINKS Center ADVANCED Session

Recommend

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data Simple methods for dealing with missing data Single and multiple imputation R example Missing data is a complex problem We must consider:

377 views • 22 slides
Estimands and Missing Data Working Group ISCTM Spring Meeting Feb 19, 2019 Outline

Estimands and Missing Data Working Group ISCTM Spring Meeting Feb 19, 2019 Outline

Estimands and Missing Data Working Group ISCTM Spring Meeting Feb 19, 2019 Outline Objectives of this working group Background: ICH E9(R1) Addendum Defining estimands for a trial Components of an estimand Examples of

435 views • 20 slides
Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES,

Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES,

WHAT ARE MISSING DATA ? HOW TO TREAT MISSING DATA ? LONGITUDINAL DATA, CAUSALITY, &amp; ETHICS Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES, Universit de Lausanne FORS SHP workshop June

1.2k views • 108 slides
Is the data missing at random? DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep

Is the data missing at random? DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep

Is the data missing at random? DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning &amp; Computer Vision Consultant Possible reasons for missing data Note (variable data eld or column in a DataFrame) Values

650 views • 44 slides
Wh y do missing v al u es e x ist ? FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN

Wh y do missing v al u es e x ist ? FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN

Wh y do missing v al u es e x ist ? FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN G IN P YTH ON Robert O ' Callaghan Director of Data Science , Ordergroo v e Ho w gaps in data occ u r Data not being collected properl y Collection

602 views • 32 slides
Bayesian Generalized linear mixed models with data missing not at random Overview: Two simple

Bayesian Generalized linear mixed models with data missing not at random Overview: Two simple

Bayesian Generalized linear mixed models with data missing not at random Overview: Two simple introductory examples of data missing not at random (MNAR) Missing mechanism and likelihood in the case of missing at random (MAR) as defined by

722 views • 40 slides
Eliciting and using expert opinions about informatively missing outcome data in clinical trials

Eliciting and using expert opinions about informatively missing outcome data in clinical trials

Eliciting and using expert opinions about informatively missing outcome data in clinical trials Ian White MRC Biostatistics Unit, Cambridge, UK Bayes working group German Biometric Society Kln, 3 December 2004 1 Why do Bayesian analyses?

1.13k views • 73 slides
N 1 N 1 IV x i y i s i z i s i z i (2) 3. Imputation i 1

N 1 N 1 IV x i y i s i z i s i z i (2) 3. Imputation i 1

A Course in Applied Econometrics 1 . When Can Missing Data be Ignored ? Lecture 18 : Missing Data Linear model with IVs: y i x i u i , (1) Jeff Wooldridge IRP Lectures, UW Madison, August 2008 where x i is 1 K , instruments z

236 views • 10 slides
Lecture #12: kNN Classification and Missing Data Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #12: kNN Classification and Missing Data Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #12: kNN Classification and Missing Data Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline ROC Curves k -NN Revisited Dealing with Missing Data Types of

797 views • 67 slides
Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1.

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1.

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1. Missing Data and Missing Data Mechanisms 2. Imputation 3. Missing Data and Multiple Imputation in Baseline KLoSA Data Missing Data and Multiple

634 views • 35 slides
Estimating Gaussian Mixture Models from Data with Missing Features by Daniel McMichael CSSIP

Estimating Gaussian Mixture Models from Data with Missing Features by Daniel McMichael CSSIP

Estimating Gaussian Mixture Models from Data with Missing Features by Daniel McMichael CSSIP Missing Data In classification we frequently see to classify objects using vectors of measured features. Sometimes these features are missing: [ 0

478 views • 10 slides
Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing

Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing

DataCamp Dealing With Missing Data in R DEALING WITH MISSING DATA IN R Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing Data in R Lesson overview Using imputations to understand data structure

647 views • 53 slides
Missing Data in Machine Learning Guy Van den Broeck Emerging Challenges in Databases and AI

Missing Data in Machine Learning Guy Van den Broeck Emerging Challenges in Databases and AI

Computer Science Reasoning about Missing Data in Machine Learning Guy Van den Broeck Emerging Challenges in Databases and AI Research (DBAI) Nov 12 2019 Outline 1. Missing data at prediction time a. Reasoning about expectations b.

1.91k views • 46 slides
Big data: What are you missing? the risks of assuming data equals all David J. Hand Imperial

Big data: What are you missing? the risks of assuming data equals all David J. Hand Imperial

Big data: What are you missing? the risks of assuming data equals all David J. Hand Imperial College London and Winton Capital Management 6 th January 2016 Theory of Big Data, UCL 1 BACKGROUND The promise of big data: McKinseys big data

679 views • 48 slides
Effective Missing Data Prediction for Collaborative Filtering Hao Ma, Irwin King, and Michael R.

Effective Missing Data Prediction for Collaborative Filtering Hao Ma, Irwin King, and Michael R.

Outline Introduction Missing Data Prediction Empirical Analysis Conclusions and Future Work Effective Missing Data Prediction for Collaborative Filtering Hao Ma, Irwin King, and Michael R. Lyu Department of Computer Science and Engineering

1.2k views • 118 slides
Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 ,

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 ,

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 , Ihab F. Ilyas 1 Theodoros Rekatsinas 2 1 2 Problem Missing data is a persistent problem in many fields Sciences Data mining

322 views • 28 slides
Approaches to imputing missing data in complex survey data Christine Wells, Ph.D. IDRE UCLA

Approaches to imputing missing data in complex survey data Christine Wells, Ph.D. IDRE UCLA

Introduction Offerings Examples Results Conclusions Ending Approaches to imputing missing data in complex survey data Christine Wells, Ph.D. IDRE UCLA Statistical Consulting Group July 27, 2018 Christine Wells, Ph.D. Imputing missing

1.03k views • 28 slides
Searching for and replacing missing values Nicholas Tierney Statistician DataCamp Dealing With

Searching for and replacing missing values Nicholas Tierney Statistician DataCamp Dealing With

DataCamp Dealing With Missing Data in R DEALING WITH MISSING DATA IN R Searching for and replacing missing values Nicholas Tierney Statistician DataCamp Dealing With Missing Data in R What we are going to cover How to look for hidden

468 views • 29 slides
Methodological developments for combining data Modelling non-random missing data in longitudinal

Methodological developments for combining data Modelling non-random missing data in longitudinal

Motivation Building a Bayesian joint model which combines data Results Summary Methodological developments for combining data Modelling non-random missing data in longitudinal studies: how can information from additional sources help? Alexina

592 views • 40 slides
A Course in Applied Econometrics 1. Introduction Lecture 10 2. Example I: Missing Data 3.

A Course in Applied Econometrics 1. Introduction Lecture 10 2. Example I: Missing Data 3.

Outline A Course in Applied Econometrics 1. Introduction Lecture 10 2. Example I: Missing Data 3. Example II: Returns to Schooling Partial Identification 4. Example III: Initial Conditions Problems in Panel Data 5. Example IV: Auction

128 views • 8 slides
Prediction from low-rank missing data Elad Hazan Roi Livni Yishay Mansour Princeton U Hebrew U

Prediction from low-rank missing data Elad Hazan Roi Livni Yishay Mansour Princeton U Hebrew U

Prediction from low-rank missing data Elad Hazan Roi Livni Yishay Mansour Princeton U Hebrew U Tel-Aviv U &amp; Microsoft Research (all of us) Recommendation systems Predicting from low-rank missing data Gender? Annual income? Will buy

400 views • 25 slides
Missing-data masks in all-combinations multi-band decoding It is shown that for MAP decoding

Missing-data masks in all-combinations multi-band decoding It is shown that for MAP decoding

morris@idiap.ch http://www.idiap.ch/ Missing-data masks in all-combinations multi-band decoding It is shown that for MAP decoding with all-comb experts multi-band expert weighting can make use of same soft missing-data mask as used with

477 views • 5 slides
Handling missing data P R AC TIC IN G MAC H IN E L E AR N IN G IN TE R VIE W QU E STION S IN

Handling missing data P R AC TIC IN G MAC H IN E L E AR N IN G IN TE R VIE W QU E STION S IN

Handling missing data P R AC TIC IN G MAC H IN E L E AR N IN G IN TE R VIE W QU E STION S IN P YTH ON Lisa St u art Data Scientist Prereq u isites S u per v ised Learning w ith scikit - learn Uns u per v ised Learning in P y thon PRACTICING

594 views • 33 slides
Dealing with Missing Data Challenges and Solutions Nicole Erler Department of Biostatistics,

Dealing with Missing Data Challenges and Solutions Nicole Erler Department of Biostatistics,

Dealing with Missing Data Challenges and Solutions Nicole Erler Department of Biostatistics, Erasmus Medical Center n.erler@erasmusmc.nl N_Erler www.nerler.com NErler 13 January 2020 Handling Missing Values is Easy! Functions

1.12k views • 89 slides

More recommend