Recommended Practices for the design of business surveys questionnaires Stefania Macchia macchia@istat.it (Istat, National Accounts and Economic Statistics Department) NTTS 2015 Conference Bruxelles, 10-12 March
The context ISTAT Business Statistical Portal implements a new approach for the organization and management of data collection processes. It constitutes a single entry point for web- based data collection from enterprises according to a “ business-centric ” perspective. Process 1 Businesses register Statistical Portal Communicatio n Process n Business Data capturing es systems Main Features: High integration among all the phases of the productive processes: survey design, data collection, frame, design of electronic questionnaires , elaboration, estimations, informative «return» Rationalization and intensification of the informative flows among «stakeholders» involved in statistical processes (production lines, statistical registers, data capturing systems, dissemination and communication, statistical units) Introduction of new management procedures characterised by high interrelation degree among Portal, production lines, production support facilities High potential in terms of systemic efficiency 2
Harmonisation of business surveys questionnaires The new Business Portal architecture implies the acceleration of the harmonization of survey questionnaires, both in terms of concepts treated and features of electronic questionnaires, with the perspective of: reducing respondent burden in terms of duplication of information requested adopting a common language of communication with enterprises through a standard approach for questionnaires design, which takes into consideration the peculiarities of the different business surveys questionnaires (SBS and STS), with attention to the variables definitions management and to the structure of questions to collect information enhancing the level of quality (in terms of response rates, but also homogeneity of collected data). Manual of Recommended Practices for the design of business surveys questionnaires 3
Manual of Recommended Practices for the design of business surveys questionnaires Not all the aspects of the structured process to design questionnaires have been treated because, in this context, the aim was not to plan new surveys ex novo, but to restructure and harmonize surveys to be included in Business Portal. The manual is concentrated on: The definition of the questionnaire schemes All the aspects of the questionnaire design related to the data capturing technique GSBPM v.5: Questionnaire Electronic Questionnaire 2.2 Design variable descriptions design design 2.3 Design collection ‘The sub -process 2.2 may need to run in parallel with sub- process 2.3, as the definition of the variables to be collected, and the choice of collection instrument may be inter- dependent’ 4
Recommended Practices for the design of business surveys questionnaires In defining Recommended Practices, it has been considered how some aspects of the design of questionnaires and of the features to be implemented in the electronic questionnaire have an impact on the steps of the response process, represented in a ‘ hybrid response model’ based on a cognitive approach. The questions wording , their structuring , the layout and all the instructions and information at disposal Encoding in Memory / Record Formation of respondents related to each question impact on Selection / Identification of Respondent(s) the comprehension and retrieval processes. Assessment of Priorities (Motivation) The skipping and consistency rules implemented in Comprehension electronic questionnaire support for the judgement Retrieval step Judgement the electronic questionnaire design impact on the Communication . communication process , as it predefines the structures according to which responses have to be Release of the Data given. the questionnaire design impact on the Encoding in memory/Record formation step because as more Ibrid response model (2000 Sudman, 2001 Wilimack, the way data are stored in the respondent data base is Nichols) considered in structuring questions, as more the effort needed to provide data is contained. 5
Manual of Recommended Practices for the design of business surveys questionnaires First section of the Manual: General principles and particular ones related to the CAWI technique (shortly described) Principles defined according to a strategy which should: Encourage responses Contain responses errors Reduce the effort necessary to provide answers Give respondents a feedback Very important general principle, consistent with an integrated approach: contain information to be collected, using sources already available like: Administrative/fiscal sources Data collected in other surveys Official registers. 6
Manual of Recommended Practices The set of recommendations provided are organised in paragraphs according to specific homogeneous themes: 1) Design of questionnaire sections/Web pages Suggestions are given on how designing the sections, taking into considerations the Web pages, which in turn, can be composed by more than one computer screen-shots. 2) Questions design Identification of the structure to be used to provide the answer: suggestions are provided on quantitative questions (how to manage decimal values, scale factors, measure units, etc.), closed questions (when using radio-buttons, drop-down-boxes, check-boxes), free texts questions to be coded (when it is preferred to use assisted coding functions) and, finally, questions organised in tables/arrays. 3) Use of classifications When classifications are used to classify textual responses, suggestions are given on different algorithms for textual matching which can be used. 7
Manual of Recommended Practices 4) The structure for variables definitions Harmonising the definitions in terms of their content is surely the first problem to be faced but how this content is expressed and structured has in impact on the comprehension process as well. For this reason a common modular structure for definitions of variables has been defined 5) The management of skipping rules The automatic management of skipping rules is a strength of data collection computer assisted techniques. Concerning business surveys questionnaires, different suggestions are given for skipping rules related to questions belonging to the same section/web page or to different section/web pages. 6) The checking plan Suggestions are given on the management of the consistency rules, taking into consideration aspects, like: rules regarding variables of the same web-page or different ones, number of checking rules on a single web-page, relation between skipping and checking rules, etc. 7) The screen design Standards already defined for software package used in the Business Portal architecture have been adopted. Some details have been provided on graphic symbols to be used to link definitions, instructions, helps, etc. 8
Manual of Recommended Practices 8) The guide and instructions for respondents A set of documents is always provided to respondents (e.g. brief description of the survey, normative aspects concerning confidentiality, compilation guide, etc.). Common structures, templates and access modes have been designed for these documents, because they can have an impact on the comprehension process as well. 9) Questions to estimate the perceived respondent burden The migration of surveys questionnaires in the new generalised sw system for electronic questionnaires implementation is a good occasion to collect information on respondent burden directly from respondents. For this purpose a set of questions has been proposed, which distinguish possible burden due to difficulty in retrieving the requested information and in filling in the questionnaire. The responses to these questions could be analysed together with other indications derivable from paradata. 10) The use of paradata for data collection monitoring and optimising questionnaire design The Business Portal already produces some indicators for the monitoring purpose. A set of other indicators has been identified which can be used by the researcher to optimise the questionnaire design, specifically when some ‘alerts’ are highlighted by the answers given to questions on the respondent burden (e.g. number of entries for web-pages, connection time for the entire questionnaire or for each web-page, length of navigation routings, etc.). 9
The structure for variables definitions The harmonisation of the contents to be observed In business statistics some variables can be collected in different surveys questionnaires. This can depend on different informative needs (structural and short- term surveys), while in other situations it could sometime result to be redundant. On the other hand, big business agencies are often involved in more surveys. Definitions of the same variables could be different: in terms of their content because EU regulations contain different specifications for structural and short-term surveys, or in terms of wording, because expressed differently by each survey responsible. Contents harmonisation in terms of variables definitions and questions wording is the pre-requirement for a two steps activity: Improvement of the harmonisation and comparability of collected data Reduction of redundancies and respondent burden 10 10
Recommend
More recommend
