Missing data due to a checklist-effect Dr. Thorsten Meyer 1) , Dr. - PowerPoint PPT Presentation

Institut für Sozialmedizin Campus Lübeck Missing data due to a ‘checklist-effect’ Dr. Thorsten Meyer 1) , Dr. Ines Schäfer 1) , Dr. Christine Matthis 1) Prof. Dr. Thomas Kohlmann 2) ; Dr. Oskar Mittag 1) 1) Institute for Social Medicine, University Clinics Schleswig-Holstein, Campus Luebeck 2) Institute for Community Medicine, Ernst-Moritz-Arndt-University, Greifswald acknowledgement ‘Die Abschätzung von Rehabedarf bei aktiven Mitgliedern der Gesetzlichen Rentenversicherung: der Lübecker Algorithmus und seine Validierung’, A1-project, principal investigator: Prof. Dr. Dr. H. Raspe, Norddeutscher Verbundes für Rehabilitationsforschung, supported by BMBF and VDR (FKZ: 02 1 06) Folie 1 / 15

Problem… Institut für Sozialmedizin Campus Lübeck Folie 2 / 15

…and possible (simple-minded) solutions: Institut für Sozialmedizin Campus Lübeck according to manual: Som = Missing imputation by group mean: Som = 1.15 imputation by individual mean: Som = 1.60 assumption of a „checklist effect“: Som = 0.67 Folie 3 / 15

assumption: checklist-effect Institut für Sozialmedizin Campus Lübeck definition by the following response pattern: (1) at least one missing value, and (2) at least one valid item response, and (3) no ‘not at all’-responses. In these respondents, all items with missing data in the respective scale are interpreted as „not at all“ -responses. Folie 4 / 15

Number of missing item responses in relation to the number of valid positive responses (‘checklist effect’ highlighted in the diagonal) Institut für Sozialmedizin number of missing item responses Campus Lübeck 1 2 3 4 5 6 7 8 9 10 11 12 0 0 0 0 0 0 0 0 0 0 0 0 0 somatisation 1 0 0 0 0 0 0 0 0 0 0 4 subscale of 2 0 0 0 0 0 0 0 0 0 4 SCL-90-R 3 0 1 1 0 0 0 0 1 6 4 3 2 0 0 0 0 0 4 number of valid positive 5 2 0 0 0 0 0 6 responses without 6 1 0 0 0 0 3 “not-at-all”- 7 1 3 0 0 0 responses 8 2 0 1 1 9 2 0 1 10 0 3 11 5 Folie 5 / 15

Sample • n=228 Institut für Sozialmedizin • primarily blue collar workers who previously Campus Lübeck had filed applications for medical rehabilitation benefits (esp. due to back pain) • postal survey age 50,1 (SD=6,5) male sex 73,2 % (167) family status single 11,0 % (25) married 75,4 % (172) divorced / separated 11,4 % (26) widowed 2,2 % (5) education secondary school (Hauptschule) 78,1 % (178) secondary school (Realschule) 11,4 % (26) job full-time 68,9 % (153) work status blue-collar / manual worker 85,8 % (194) white-collar / clerical worker 12,4 % (28) Folie 6 / 15

1. prevalence of the checklist-effect? 1. prevalence of the checklist-effect? 2. stable phenomenon within the questionnaire? Institut für Sozialmedizin Campus Lübeck somatisation subscale SCL-90-R 75% 16% 0% 9% no missing data checklist-effect other types of missing data no valid item responses at all Folie 7 / 15

succeeding items: questions on pain loci Institut für Sozialmedizin Campus Lübeck Folie 8 / 15

depression (CES-D) Institut für Sozialmedizin Campus Lübeck Folie 9 / 15

1. prevalence of the checklist-effect? 2. stable phenomenon within the questionnaire? Institut für Sozialmedizin Campus Lübeck somatisation subscale pain items depressiveness SCL-90-R CES-D 87% 83% 75% 16% 2% 11% 10% 6% 1% 0% 0% 9% no missing data • of those subjects with a checklist-effect in the somatisation subscale, 7 out of 10 had a checklist-effect checklist effect in the pain items other types of missing data • all subjects with a checklist-effect in the pain no valid item responses at all items had a checklist-effect in the somatisation subscale, too Folie 10 / 15

3. Do subjects with the postulated checklist-effect differ from the other subjects with regard to social characteristics and health status? Institut für Sozialmedizin type of missing data test on difference Campus Lübeck No. between groups „checklist other no missing statistic, -effect“ missing data degrees of freedom, (n=37) data (n=171) level of significance, (n=20) effect size χ 2 =2.9; df=2 Sex male 30 12 125 (81.1 %) (60.0 %) (73.1 %) p=.229, V C =.11 age M=49,8 M=50,3 M=52,0 F=0.27; df betw =2; SD=6,3 SD=6,7 SD=7,6 df within =220; p=.36 η 2 =0.009 χ 2 =3.7; df=2 education lower a) 148 b) 28 15 (75.7 %) (75.0 %) (86.5 %) p=.154; V C =.13 medium or higher c) 9 5 23 (24.3 %) (25.0 %) (13.5 %) χ 2 =3.395; df=6; health status good 0 1 2 p=.758; V C =.09 (5.0 %) (1.2 %) satisfactory 7 4 30 (20.6 %) (20.0 %) (17.6 %) not good 20 11 110 (64.7 (58.8 %) (55.0 %) %) bad 7 4 28 (20.6 %) (20.0 %) (16.5 %) vitality M=36.6 M=37.8 M=39.0 F=0.27; df betw =2; (SF-36, 0-100) SD=19.9 SD=21.2 SD=17.9 df within =212; p=.76 η 2 =0.002 depression M=19.9 M=16.3 M=17.1 F=1.44; df betw =2; (CES-D, 0-60) SD=10.6 SD=7.7 SD=8.8 df within =218 ;p=.239 η 2 =0.013 Folie 11 / 15

4. results of different imputation procedures Institut für Sozialmedizin Campus Lübeck imputation manual- manual- ML- checklist- based based + estimation effect group mean (max. 4 MD) (EM-algorythm) N valid 200 228 228 227 N missing 28 0 0 1 mean 1.04 1.04 0.96 0.98 sd 0.67 0.62 0.58 0.61 M=.65 SD=.37 N=27 Folie 12 / 15

5. different covariance structures? Institut für Sozialmedizin Campus Lübeck imputation manual- manual- ML- checklist- based based + estimation effect group mean (EM- (max. 4 MD) algorythm) depression .531 .504 .536 .496 (CES-D) n=190 n=215 n=215 n=215 rumination .437 .425 .436 .446 (PRSS) n=193 n=216 n=216 n=215 functional capacity -.402 -.373 -.413 -.391 (FFbH-R) n=200 n=226 n=226 n=225 vitality -.335 -.317 -.370 -.304 (SF-36) n=195 n=222 n=221 n=221 Pearson correlation coefficient (all r p<.001) Folie 13 / 15

checklist-effect in other surveys Institut für Sozialmedizin Campus Lübeck 35 30,6 30 25 Checkliste 20 16,2 Sonstige MD 13,9 15 8,8 8,7 10 5 1,4 0 A4-Survey Rehaantragsteller Kurgäste . Angaben in Prozent Folie 14 / 15

Discussion and conclusions Institut für Sozialmedizin 1. assumption of checklist effect accomplished an almost Campus Lübeck complete imputation of missing values based on theory 2. reduction of bias towards inclusion of less ill subjects 3. the missingness does not seem to be conditional on some other variable(s) observed in the data (MAR) 4. ML-estimation yielded similar mean and sd, but different correlations � identification of checklist-effect in other samples � analyzing validity in methodological studies, e.g. cognitive survey Suggestion: Identification of possible checklist-effects in questionnaires + if present: additional coding of missing values as non-affirmative responses Folie 15 / 15