Gov 51: Missing Data Matthew Blackwell Harvard University 1 / 7

Civilian attitudes and war against insurgency • War in Afghanistan: counter-insurgency war • Military against insurgents • Key to victory: winning hearts and minds of civilians • Aid provision, information campaign, minimizing civilian casualties • How does exposure to violence afgect support for Taliban, coalition? 2 / 7

Afghan study 0 Logar Baraki Barak 80 18 10 ## employed income violent.exp.ISAF ## 1 0 2,001-10,000 0 ## 2 1 2,001-10,000 ## 3 12 1 2,001-10,000 1 ## 4 1 2,001-10,000 0 ## 5 1 2,001-10,000 0 ## 6 1 <NA> 0 ## 6 21 afghan <- read.csv(”data/afghan.csv”) 49 head(afghan[, 1:8]) ## province district village.id age educ.years ## 1 Logar Baraki Barak 80 26 10 ## 2 Logar Baraki Barak 80 3 80 ## 3 Logar Baraki Barak 80 60 0 ## 4 Logar Baraki Barak 80 34 14 ## 5 Logar Baraki Barak 3 / 7

• Missing data in R: a special value NA Missing data • Nonresponse : respondent can’t or won’t answer question. • Sensitive questions social desirability bias • Some countries lack offjcial statistics like unemployment. • Leads to missing data. • Causes problems with calculating statistics: ## prop. of those who got hurt by ISAF mean(afghan$violent.exp.ISAF) ## [1] NA 4 / 7

• Missing data in R: a special value NA Missing data • Nonresponse : respondent can’t or won’t answer question. • Some countries lack offjcial statistics like unemployment. • Leads to missing data. • Causes problems with calculating statistics: ## prop. of those who got hurt by ISAF mean(afghan$violent.exp.ISAF) ## [1] NA 4 / 7 • Sensitive questions ⇝ social desirability bias

Missing data • Nonresponse : respondent can’t or won’t answer question. • Some countries lack offjcial statistics like unemployment. • Leads to missing data. • Causes problems with calculating statistics: ## prop. of those who got hurt by ISAF mean(afghan$violent.exp.ISAF) ## [1] NA 4 / 7 • Sensitive questions ⇝ social desirability bias • Missing data in R: a special value NA

Handling missing data in R • Adding na.rm = TRUE to some functions removes missing data. mean(afghan$violent.exp.ISAF, na.rm = TRUE) ## [1] 0.375 • Or, you can explicitly remove missing values using na.omit() function: mean(na.omit(afghan$violent.exp.ISAF)) ## [1] 0.375 • Add NA to table() with exclude = NULL : table(ISAF = afghan$violent.exp.ISAF, exclude = NULL) ## ISAF ## 0 1 <NA> ## 1706 1023 25 5 / 7

• Or, you can explicitly remove missing values using na.omit() function: Handling missing data in R • Adding na.rm = TRUE to some functions removes missing data. mean(afghan$violent.exp.ISAF, na.rm = TRUE) ## [1] 0.375 mean(na.omit(afghan$violent.exp.ISAF)) ## [1] 0.375 • Add NA to table() with exclude = NULL : table(ISAF = afghan$violent.exp.ISAF, exclude = NULL) ## ISAF ## 0 1 <NA> ## 1706 1023 25 5 / 7

Handling missing data in R • Adding na.rm = TRUE to some functions removes missing data. mean(afghan$violent.exp.ISAF, na.rm = TRUE) ## [1] 0.375 • Or, you can explicitly remove missing values using na.omit() function: mean(na.omit(afghan$violent.exp.ISAF)) ## [1] 0.375 • Add NA to table() with exclude = NULL : table(ISAF = afghan$violent.exp.ISAF, exclude = NULL) ## ISAF ## 0 1 <NA> ## 1706 1023 25 5 / 7

• Add NA to table() with exclude = NULL : Handling missing data in R • Adding na.rm = TRUE to some functions removes missing data. mean(afghan$violent.exp.ISAF, na.rm = TRUE) ## [1] 0.375 • Or, you can explicitly remove missing values using na.omit() function: mean(na.omit(afghan$violent.exp.ISAF)) ## [1] 0.375 table(ISAF = afghan$violent.exp.ISAF, exclude = NULL) ## ISAF ## 0 1 <NA> ## 1706 1023 25 5 / 7

Handling missing data in R • Adding na.rm = TRUE to some functions removes missing data. mean(afghan$violent.exp.ISAF, na.rm = TRUE) ## [1] 0.375 • Or, you can explicitly remove missing values using na.omit() function: mean(na.omit(afghan$violent.exp.ISAF)) ## [1] 0.375 table(ISAF = afghan$violent.exp.ISAF, exclude = NULL) ## ISAF ## 0 1 <NA> ## 1706 1023 25 5 / 7 • Add NA to table() with exclude = NULL :

Gov 51: Missing Data Matthew Blackwell Harvard University 1 / 7 - PowerPoint PPT Presentation

Gov 51: Missing Data Matthew Blackwell Harvard University 1 / 7 Civilian attitudes and war against insurgency War in Afghanistan: counter-insurgency war Military against insurgents Key to victory: winning hearts and minds of

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data

Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES,

Is the data missing at random? DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep

Wh y do missing v al u es e x ist ? FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN

Bayesian Generalized linear mixed models with data missing not at random Overview: Two simple

N 1 N 1 IV x i y i s i z i s i z i (2) 3. Imputation i 1

Lecture #12: kNN Classification and Missing Data Data Science 1 CS 109A, STAT 121A, AC 209A,

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1.

Estimating Gaussian Mixture Models from Data with Missing Features by Daniel McMichael CSSIP

Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing

Missing Data in Machine Learning Guy Van den Broeck Emerging Challenges in Databases and AI

Big data: What are you missing? the risks of assuming data equals all David J. Hand Imperial

Effective Missing Data Prediction for Collaborative Filtering Hao Ma, Irwin King, and Michael R.

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 ,

Approaches to imputing missing data in complex survey data Christine Wells, Ph.D. IDRE UCLA

Searching for and replacing missing values Nicholas Tierney Statistician DataCamp Dealing With

Methodological developments for combining data Modelling non-random missing data in longitudinal

A Course in Applied Econometrics 1. Introduction Lecture 10 2. Example I: Missing Data 3.

Prediction from low-rank missing data Elad Hazan Roi Livni Yishay Mansour Princeton U Hebrew U

Missing-data masks in all-combinations multi-band decoding It is shown that for MAP decoding

Handling missing data P R AC TIC IN G MAC H IN E L E AR N IN G IN TE R VIE W QU E STION S IN

Dealing with Missing Data Challenges and Solutions Nicole Erler Department of Biostatistics,

Testing and Imputing Item Nonresponse as Missing Data, with Big and Normal Survey Data NATALIE

Missing-data methods with cepstral data Summary of work done at IDIAP Main achievements