A Multiple Method Approach to Testing a Complex Web‐based Establishment Survey Instrument Aryn Hernandez Temika Holland Rebecca Keegan Amy Anderson Riemer U.S. Census Bureau July 12, 2019 Disclaimer: Any views expressed are those of the author and not necessarily those of the U.S. Census Bureau. The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied. (Approval ID: CBDRB‐FY19‐ESMD‐B00016) 1
Overview • Economic Census Instrument Design Challenges • Economic Census Instrument and Respondent Portal Features • Multi‐year, Multi‐method Testing Approach Overview • Usability Testing Highlights • Economic Census Instrument • Respondent Portal • How‐to Videos • Ongoing and Future Research 2
The Economic Census (EC) • Every five years, referencing years ending in 2 and 7 • Mandatory, self‐administered establishment‐based survey • Requests data about nearly 4 million establishments • Massive multinational companies with 1000s of establishments, single location businesses, and everything in between • All industries • All levels of computer literacy • Collects a wide range of financial and production data • For example: payroll, employment, revenue by product type, expenses, etc… • Questions vary by industry classification • Answering the survey can involve multiple people within a business to complete and typically requires access to business records • Data are used for economic benchmarks and by policy makers, federal agencies, trade associations, businesses, and others • Between 2012 and 2017, the survey instrument for the EC was radically transformed from a downloaded software application to an entirely web‐based platform 3
The Centurion Instrument • Retains much of the downloadable software’s functionality, but… • Entirely web‐based • Cleaner, more modern • More obvious navigation • Clearly defined reporting steps Any company data shown is dummy data. 4
Design Challenges (cont.) • Concurrent with the development of the Centurion Instrument • Issues with current economic survey methods: • Assigning individual userIDs and passwords for each survey • Different URL for each survey • Evolving Security Standards • Solution: • One URL for all surveys • One set of login credentials 5
Any company data shown is The Respondent Portal dummy data. 6
Before Portal After Portal The Respondent Portal • Respondents… • must create their own accounts • must link the survey to their accounts Multiple Multiple Survey Survey Survey Survey Requests Requests Requests Requests • can view survey Information • can request extensions Business Business Respondent Respondent • can access secure messaging center • can share survey access with others in Multiple One organization Websites, Websites, Portal, Portal, Multiple Multiple One Login One Login Logins Logins 7
Multi‐Year, Multi‐Method Testing Overview • Methodologies: • Task analysis • Cognitive testing • Usability testing • Paper prototypes, high‐fidelity prototypes • Contact strategy experiments • Respondent debriefings • Paradata evaluations 8
Usability Testing Highlights • Evaluated the web instrument and portal to identify issues and assess the site’s ease of use • In person, at respondents’ place of business • Respondents provided feedback on components of the 2017 EC electronic reporting system: 1. CPASS (Respondent Portal registration process) 2. Respondent Portal Prototype 3. Centurion prototype 9
Usability Testing Highlights • Methodology: • Metrics: • Introductions/Informed Consent • Effectiveness • Demographic Questionnaire • Percentage of successful completions for each task • User tasks • Efficiency • Think aloud • Task completion time • Probes • Subjective satisfaction • Satisfaction survey (SUS) • Participants’ satisfaction ratings • Debriefing • Verbal and non‐verbal behavior 10
• Metrics: Usability Testing Highlights • Effectiveness • Efficiency • Subjective satisfaction • Centurion User Tasks: • Verbal and non‐verbal 1. Logging into the survey behavior 2. Updating locations lists (Add/Delete Locations) 3. Locating printable worksheet 4. Beginning reporting 5. Downloading/Uploading the spreadsheet (Multi‐locations only) 6. Reviewing entered data and resolving issues 7. Submitting the completed survey 11
• Metrics: Usability Testing Highlights • Effectiveness • Modified System Usability Scale • Efficiency • Subjective satisfaction 1. I think that I would like to use the site to complete the Economic Census • Verbal and non‐verbal 2. I found the site to be simple behavior 3. I thought the site was easy to use 4. I think that I could use the site without the support of a technical person 5. I found that the various functions in the site were well put together 6. I thought there was a lot of consistency in the site 7. I would imagine that most people would learn to use the site very quickly 8. I found the site to be very intuitive 9. I felt very confident using the site 10. I could use the site without having to learn anything *Multi‐location respondent results (n=29) 12
Usability Testing Highlights • Respondent Portal Tasks: 1) Create an account 2) Link a survey using authentication code 3) Locate and use the ‘Share Survey’ feature 4) Locate survey information 5) Locate the ‘Help’ button 6) Locate the secure messaging form 7) Request an extension 8) Check filing status 13
How‐To Videos • Usability testing, debriefings, and paradata analysis revealed that respondents do not always use available tools and features in the instrument • To modernize our support and enhance user experience, we tested short 'How‐to' videos for specific features and user actions • Topics for these videos were chosen from the most difficult user tasks from usability testing as well as the most common call center questions • About 2 minutes or less in length • Screen recordings with narration Video Topics 1. Dashboard Overview 6. Messages Feature 2. Organizing Dashboard 7. Upload Spreadsheet 3. Print Preview 8. Reviewing Data 4. Add/Delete 9. Submitting Survey 5. Download Spreadsheet 10. Creating a Portal Account 14
How‐To Video Testing • Respondents completed tasks similar to usability testing • Respondents then viewed and rated the how‐to videos • Finally, debriefing questions were asked • Included internet experience questionnaire asking about average usage of how‐to videos • Videos were well liked and thought to be useful to future respondents • Selected results from Internet Experience Questionnaire: • How often do you seek out how‐to videos when you need assistance with a task? • Average response: Somewhat often (4.3/7) • How likely would you be to access a how‐to video if you’re learning a new skill? • Average response: Pretty likely (5.3/7) • Respondents reported watching an average of 1 ‐2 help videos over the past month. 15
Ongoing and Future Testing • Just because the 2017 Economic Census data collection period has ended does not mean testing ends! • Centurion Instrument is still the home of ASM/COS • More surveys being moved into Respondent Portal • Looking towards 2022! • Starting to include more demographic questions in our interviews to get a better understanding of our actual end users and not just the companies themselves • Continuing to analyze paradata to see respondent behaviors outside of an interview setting • Incorporating eye‐tracking 16
Thank you! Contact Info Temika Holland Aryn Hernandez Temika.Holland@census.gov Aryn.Hernandez@census.gov 301‐763‐5241 301‐763‐7982 Amy Anderson Riemer Rebecca Keegan Amy.E.Anderson.Riemer@census.gov Rebecca.Keegan@census.gov 301‐763‐7544 301‐763‐6003 17
Recommend
More recommend
