7 th Workshop on Labour Force Survey Methodology Calculating the eligibility rate of sampling units with unknown eligibility Rita Lima Rita Ranaldi lima@istat.it ranaldi@istat.it Italian National Statistical Institute Madrid, 11 May 2012
Classification of units according to eligibility A core issue is the definition of “eligible unit” as it affects the calculation of response rate 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Definition of “eligible unit” in the IT LFS A unit is eligible : - if the name corresponds to the selected household and - if it is a private household having usual residence in the municipality The eligibility is unknown when it is not possible to collect sufficient information for a proper classification, e.g. � no contact made during fieldwork period � no one at home and no other information available (CAPI) � unreachable due to wrong telephone number (CATI) 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Units according to eligibility (shares) IT LFS, by survey mode and wave - 1 st quarter 2011 0% 20% 40% 60% 80% 100% T 5,9 Total O % of units with unknown eligibility Wave 1 6,8 T A Wave 2-4 5,4 L Total 1,9 C A 2,4 Wave 1 P 1,5 Wave 2-4 I 9,3 Total C A Wave 1 15,9 T 7,5 Wave 2-4 I Eligible Not eligible Unknown eligibility 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Response rate - one of the most important quality indicators for the social sample surveys - used between surveys, years and countries to compare survey quality EI = × RR 100 + + α ⋅ EI EN UN where: EI Number of eligible interviews EN Number of eligible non-interviews UN Number of units with unknown eligibility α Estimated proportion of units of unknown eligibility that are actually eligible 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Estimating Eligibility Rates: a review � The Minimum and Maximum Allocation method (MMA) � The Proportional Allocation method (CASRO) � The American Association for Public Opinion Research (AAPOR) approach � The Survival Analysis method (SAM) 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
The Minimum and Maximum Allocation method (MMA) • Defines the lower and upper boundaries of the response rate: all units with unknown status are actually eligible (upper) or non-eligible (lower) • So it is possible to define a range of response rates by setting: α = α = 0 1 MMA MMA upper lower 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
The Proportional Allocation method (CASRO) • Proportion of eligible units amongst those whose eligibility is unknown = Proportion of eligible units amongst the eligibility known sample units • CASRO formula for α is: + EI EN α = CASRO + + EI EN NE where: EI Number of eligible interviews EN Number of eligible non-interviews NE Number of not eligible units 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
The American Association for Public Opinion Research (AAPOR) approach • Estimation of the eligibility rate left to discretion of researchers − on the basis of the best available scientific information − the basis of the estimate must be explicitly stated and explained − if no information available, all units of unknown eligibility should be considered as eligible • AAPOR formula for α is: α = 1 AAPOR 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
The Survival Analysis method (SAM) • Estimates the eligibility rate by modeling the “time to resolution (death)” of each sampling units using survival analysis. It uses the additional information on the number of attempts until resolution (eligible or ineligible). • Being resolved as eligible or ineligible is comparable to two different “causes of deaths”. ˆ × − ˆ S R n n ( 0 ) ( ) ∞ = eligible ˆ α = tot eligible R ∞ SAM + ˆ ˆ S S n ( 0 ) ( 0 ) eligible ineligible unknown where: where: Ŝ eligible (0) n tot total sample size survival function for resolving as eligible n eligible number of eligible units Ŝ ineligible (0) survival function for resolving as n unknown number of units with unknown not eligible eligibility 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
An application from IT-LFS: main results /1 Ranges of the response rates according to MMA method in the IT LFS by survey mode and wave - 1 st quarter 2011 60,0 65,0 70,0 75,0 80,0 85,0 90,0 95,0 100,0 Total T 82,5 87,7 O Wave 1 69,8 75,1 T A Wave 2.4 89,0 94,0 L Total C 80,0 81,7 A Wave 1 73,2 71,3 P I Wave 2.4 88,3 89,7 84,6 Total 93,2 C A Wave 1 66,7 79,4 T I 89,3 Wave 2.4 96,5 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
An application from IT-LFS: main results /2 Eligibility rate for CATI mode of the IT-LFS according to SAM method by wave and number of call attempts – 1 st quarter 2011 waves 2-4 1 wave 1 1 0.9 0.9 0.8 0.8 Elegibility Rate Elegibility Rate 0.7 0.7 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0 5 10 15 20 27 34 50 0 10 20 30 40 50 60 70 80 94 107 123 Number of call attempts Number of call attempts The main result is that, insisting endlessly to contact The main result is that, insisting endlessly to contact the units, at the end they are almost all eligible the units, at the end they are almost all eligible 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
An application from IT-LFS: main results /3 Estimated eligibility rates and response rates according to different methods in the IT CATI LFS – 1 st quarter 2011 Eligibility rate ( α ) Response rate (RR) MMA MMA MMA MMA CASRO AAPOR SAM CASRO AAPOR SAM lower upper lower upper Total 1 0 0.9990 1 0.9990 84.558 93.210 84.566 84.558 84.566 Wave 1 1 0 0.9959 1 0.9967 66.728 79.439 66.771 66.728 66.763 Waves 2-4 1 0 0.9997 1 0.9996 89.313 96.545 89.315 89.313 89.316 There is no practical difference between the There is no practical difference between the different methods different methods 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Conclusions /1 � MMA method advantage: comparability of the results advantage: : communication issues and the range of response rate drawback : drawback could be great � CASRO method advantage: easy to apply and it does not inflate the response advantage: rate drawback: the assumption that the units with uncertain eligibility drawback: have the attributes as the units with known eligibility may be too strong � AAPOR approach advantage: easy to apply advantage: drawback: it inflates the response rate and the assumption of drawback: considering all uncertain units as eligible may be strong 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Conclusions /2 � SAM method advantage: it estimates accurately the eligibility rate if we have advantage: large sample of units that are contacted more times drawback: its application to this problem is relatively new and it drawback: is a very complicated estimation method for the current practice Under these conditions the CASRO method Under these conditions the CASRO method produces more similar results to the SAM method for produces more similar results to the SAM method for the IT LFS the IT LFS Moreover it seems the most appropriate one to Moreover it seems the most appropriate one to estimate the eligibility rate in case there is evidence estimate the eligibility rate in case there is evidence α is less than 1 that α is less than 1 that 7 th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012
Thank you for your attention
Recommend
More recommend
