1 Some initial reflections on measuring response BLUE-ETS Conference on Business Burden and Motivation in NSI Surveys burden and NSIs Statistics Netherlands, Heerlen, March 22 & 23, 2011 • Measuring response burden is important to NSIs, but why? Response burden: trends and Response burden: trends and – Response burden may influence motivation and data quality – Response burden may indicate questionnaire problems consequences consequences – Authorities, businesses and their interest organizations are concerned about businesses’ bottom line � But should we listen to them? 1, Gustav Dag Gravem 1, Dag Gravem Gustav Haraldsen Haraldsen and and Tora Tora Löfgren Löfgren • Response burden is measured in many different ways by Statistics Norway Statistics Norway NSIs, making international comparison difficult… 1 dfg@ssb.no 1 dfg@ssb.no • Often, a coherent strategy for measuring response burden and using the results is difficult to find. 1 2 Response burden measurement at Statistics Response burden measurement at Statistics Norway Norway • Like many other NSIs, Statistics Norway has not had a • …andwe do have some data that allows us to investigate systematic strategy for measuring differences and changes some of these aspects in a longitudinal perspective in response burden. – 2004: 1 paper questionnaire • Still, we have been interested in differences… – 2006: 6 web questionnaires – 2010: 12 web questionnaires – between different questionnaires • All belong to the structural business survey family of – between different modes (e.g. paper and web) questionnaires. One questionnaire per industry. – between different subsamples – before and after going from paper to web – Some questionnaire items in common, some are industry specific – before and after questionnaires undergo content/layout changes • Measured by appending a response burden questionnaire to the regular questionnaires 3 4 The instrument: the response burden 2010 list of problems questionnaire • The number of questions • Measured both time burden and perceived response • Poor visual layout burden. We focus on the perceived response burden. • Key question: ”Did you find it easy or burdensome/difficult to • Needed help to collect necessary information complete the questionnaire?” • Had to wait for information not yet available – Very easy • The web application had low usability – Quite easy • Technical problems with the web application – Neither easy nor burdensome/difficult • Complicated calculations – Quite burdensome/difficult • Mismatch between questions and available information – Very burdensome/difficult • If burdensome/difficult: follow-up question on reasons for • Difficult to judge which response alternative was the right one perceived response burden • Unclear terms or definitions 5 6 1
Perceived response burden: what are the trends The manufacturing questionnaires for our questionnaires? • Responses to the key question converted to a perceived response • Switching from paper to web does not make the response burden (prb) index. Positive figures indicate easy to complete, negative tasks easier, despite the fact that many businesses prefer figures difficult to complete. web questionnaires • The level and change in perceived response burden seems Questionnaire/industry. Ranged from most difficult to easiest in 2004/2006. 2004/2006 2010 2010-2004/06 to be correlated with business structure Construction -0,17 -0,12 0,05 – One business companies had few problems in 2004, but were hit Manufacturing, multi business (2004) -0,08 -0,15 -0,07 quite hard when the mode changed to web International sea transport 0,02 0,16 0,14 – Multi business companies had quite a lot of problems in 2004, but Transport and communication 0,09 0,13 0,04 were hit less hard when the mode changed to web Hotels and restaurants 0,1 0,12 0,02 Domestic trade 0,1 0,11 0,01 Service industry 0,21 0,2 -0,01 Manufacturing, one business (2004) 0,32 0,18 -0,14 7 8 Reasons for response burden, 2004 and 2010. Reasons for response burden, 2004 and 2010. Manufacturing questionnaire. Manufacturing questionnaires • Web related reasons are not important (!) Several establishments. • The web questionnaire rather than the web application 60 seems to be the source of the problem 50 • The differences between the 2004 and 2010 need to be 40 Percent 2004 more closely looked into 30 2010 20 – Partly a result of outsourcing revision tasks to respondents? 10 0 h p s s s s n r u t y a l c l m n e o e n t t e o i h o o l i c a h r r t i i i e i o a t i y b n m a t O s t a a o f T g m h s l e e L s c u r u u M i d c a t o e e q - t a l c f E - e n f E C . i o N p o r . s o e f N R i t a W Causes 9 10 Differences i perceived response burden The web questionnaire that changed the most: between web questionnaires in 2006 and 2010 International sea transport • Differences are small, the index more positive for 5 of 6 • 2010: questionnaire length increased significantly for a large questionnaires subsample • In 2010, the percentage using web questionnaires had • Despite this, the prb index went from 0,02 to 0,16 increased from 30-40 to 80-90 percent – 2006 sample was very small, so actual differences may be more moderate – No indication that ”followers” have more problem than early adopters • But what about the reasons for response burden? Did they of web technology – Increased computer competence? change? • From 2006 and 2010, some of the web questionnaires became shorter, some were about equally long, and some became longer. – This does not seem to have much of an impact on perceived response burden 11 12 2
Response burden reasons for the sea transport The web questionnaire that changed the most: questionnaires in 2006 and 2010. International sea transport • Five reasons are down, two are a bit up, two markedly up, 60 and two new reasons are introduced. 50 • New industry-specific items may have introduced new 40 problems Percent 2006 30 2010 – But we remember that calculations, response categories, need of 20 help and wait for information were up for the construction questionnaire 10 • Anyway, retrieval and judgment problems appear to have 0 s r t increased in importance h s s s y a l e u p n c n n m t h o l t e l i c e o a o o r i b i i t y h i i i e r n O a t m t t T o a h f a s a g s L o m s e u l c i u e u e M c t - d o r • The prb index is down, although new sources of response q l a E - t e a E e f f C c n o N i p . r . o o s N e f t R i a burden have become part of the prb cocktail. W Reasons 13 14 Summing up so far… How different filters affected the distribution of response burden reasons. All structural business questionnaires in 2010. • Increased response burden seems to go together with a greater awareness of response burden sources (manufacturing questionnaires) 50 45 • But reduced response burden does not necessarily go 40 35 together with reduced concerns about sources of response Percent 30 Mid category excluded 25 Mid category included burdens (sea transportation questionnaire) 20 Mid category only 15 • What seems to be just as important is how much the 10 5 questionnaires are changed 0 s t y l s s h p n r n u a s e o o t i c m n c e e l o h • A consciousness-raising process regarding response l i r o t i h i t i t y b n i e i a r a O t s a a h a t m o f m e L s T g o c u l s e r u u e c i t d o q - t M a e f E - a l c n f E C e i o e N r o burden reasons? Consciousness raised by… o . s N n f o t p a i W s e R – high response burden Reason – salient changes in the questionnaire – other factors 15 16 The question on response burden reasons also Perceived response burden: does it have any worked for one ”non-target” group consequences for response quality? • The ”mid category” agreed with those reporting high prb on • Respondents taking the response task seriously -> the reasons for response burden: thorough, time consuming response process -> find the task difficult – Calculations, mismatch, response categories plus need of help finding information: information retrieval and judgment! – High quality data • Still, those reporting high prb gave the clearest message – High percieved response burden • Respondents “satisficing” -> not very concerned about regarding what the problems were • We plan to ask all respondents in future prb surveys on definitions, calculations, response categories etc. -> find the task easy sources of response burden. – Low quality data – Low perceived response burden • Both categories of respondents may make errors – To what extent do they? 17 18 3
Recommend
More recommend