Experimental methods to increase online response rates in - PowerPoint PPT Presentation

Experimental methods to increase online response rates in Australian Social Surveys Stephen Cohen stephen.cohen@abs.gov.au

Context The Australian Labour Force & History Our Household Survey Innovation Panel - Panel design - Recent results - Surprising findings Some reflections

Australian Labour Force Survey (LFS) Australia Canada NZ UK US • 8 months, 8 rotation groups • Address based sample Frequency Monthly Monthly Quarterly Quarterly Monthly • 2 week reference period • Proxy reporting for all individuals Responding 26,000 56,000 15,000 40,000 74,000 in household sample size households households households households households 50,000 100,000 30,000 100,000 105,000 • Mixed mode people people people people people • First month = >50% CAPI, 20% Response ~ 92% ~ 90% ~ 86% ~ 49% ~ 86% eForm (at 2017) rates • Months 2-7 mostly CATI or eForm Rotation 8 6 8 5 8 groups • 92% response rate Population 1.5 times 19% the 2.7 times 13.3 times • Slowly falling compared the size size the size the size • No incentives to Aus

Australian Labour Force Survey (LFS) High costs Low risk appetite • LFS accounts for very large share • Rich, Highly scrutinized & of ABS collection costs. politically sensitive series • ABS facing increasing budget • Efficient sample design, sensitive pressures to shocks • Eform mode introduced in 2012 • Brittle systems & limited staff to • Seen as the primary lever to maintain them reduce costs • Only 16% eform takeup (2017) Change is necessary ... yet constrained

MPS: Current state Convert to Not acted on CAI eform Not acted on Not By Threshhold Not By Threshhold Not Converted Reminder (if before deadline) Incomplete Form Not added Converted Approach Create Login to Receive Letters + Add Access Complete Complete Survey Survey Invite / HCDF Reminder/ FAC Form Form Account SAL Account Thank You 1 Month later… (Send SAL) (Send new letters?) 1 Month later… May 2017

LFS eform response rates Event %HCDF (New RG) %eForm (NewRG, all areas) % of ALL FRHH completed by eform 0.35 1 0.9 Overall eform takeup ~22% 0.3 0.8 0.25 0.7 0.6 0.2 0.5 0.15 0.4 0.3 0.1 HCDF rates ~18% 0.2 0.05 New RG ~16% 0.1 0 0 1309 1310 1311 1312 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1701 1702 1703 1704 1705 1706 1707 2015-2017

How do we maximise How do we maximise self-initiated response? self-initiated response?

IPND to call respondents Responsive design Monthly attrition HCDF timing Optimised approach timing Improve the form Letters Clearer instructions HCDF extension Interviewers as case managers Priority post Envelope design Personalise letters with names Email & SMS reminder content & timing How do we maximise Rolling reference periods self-initiated response? XIAM secret question Revised survey sales pitch Single use signon removal Market segmentation Magic Gradual engagement Approach strategy Better survey name

The proposal: An quasi-embedded experimental program • Monthly Randomised Controlled Trials • Small number (n=700) additional households selected, monthly , commencing October 2017 • Field procedures and materials mimic LFS except for the experimental treatment that is hypothesised to increase the likelihood of self-initiated response. • Response rates compared to those of mainstream LFS , enabling ABS to identify the causal factors driving e-collection take-up. • Minimised risk to Labour Force by segregating the experimental sample and systems from mainstream MPS • Qualitative followup to contact nonrespondents & understand why they did not respond (i.e. barriers identification) • Implementation risk minimised & managed through demonstrated, experimental/scientific approach

👎

Cumulative HCDF completion rate (cumulative %) 40 35 30 25 % 20 15 10 5 0 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 HCDF Open E1710 M1710 E1711 M1711 E1712 M1712 E1802 M1802 E1803 M1803 E1804A E1804B M1804 E1806A E1806B M1806 E1807 M1807 M1809 E1812 M1812

Common, Consistent branding New envelope teaser Reduced & Clearer, Prioritised content Graphical cues + Colour emphasis Dedicated Translation website for Information extra Statement (TIS) information

New Old

30.00% Results: 25.00% • 6.2pp improvement from new 20.00% materials 15.00% • Treatment (mostly) implemented 10.00% in live LFS survey 5.00% – Comparable results being achieved 0.00% December Control Treatment

Cumulative HCDF completion rate (cumulative %) 40 35 30 25 % 20 15 10 5 0 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 HCDF Open E1710 M1710 E1711 M1711 E1712 M1712 E1802 M1802 E1803 M1803 E1804A E1804B M1804 E1806A E1806B M1806 E1807 M1807 M1809 E1812 M1812

Cumulative HCDF completion rate (cumulative %) 40 35 30 25 % 20 15 10 5 0 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 HCDF Open E1710 M1710 E1711 M1711 E1712 M1712 E1802 M1802 E1803 M1803 E1804A E1804B M1804 E1806A E1806B M1806 E1807 M1807 M1809 E1812 M1812

First reminder… …Second “overdue” reminder Tone escalation Serif font …then… Wall of text; no images

Results: 40.00% • 13pp improvement from 35.00% 30.00% second reminder 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% Control Treatment

Cumulative HCDF completion rate (cumulative %) 40 35 30 25 % 20 15 10 5 0 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 HCDF Open E1710 M1710 E1711 M1711 E1712 M1712 E1802 M1802 E1803 M1803 E1804A E1804B M1804 E1806A E1806B M1806 E1807 M1807 M1809 E1812 M1812

Cumulative HCDF completion rate (cumulative %) 40 35 30 25 % 20 15 10 5 0 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 HCDF Open E1710 M1710 E1711 M1711 E1712 M1712 E1802 M1802 E1803 M1803 E1804A E1804B M1804 E1806A E1806B M1806 E1807 M1807 M1809 E1812 M1812

10% 15% 20% 25% 30% 35% 0% 5% 1309 strategy introduced inspired” approach 1310 “Behaviourally 1311 1312 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 Event 1412 1501 1502 1503 1504 XIAM introduced HCDF (New RG FRHH) % 1505 1506 1507 1508 1509 1510 LFS eform response rates 1511 1512 1601 1602 1603 %eForm (NewRG, all areas) 1604 1605 1606 1607 1608 1609 1610 1611 1612 1701 1702 1703 % of ALL FRHH completed by eform 1704 1705 1706 1707 1708 HSIP starts 1709 1710 1711 1712 1801 1802 Tranche1 1803 HSIP 1804 1805 1806 1807 1808 1809 1810 Tranche2 1811 HSIP 1812 1901 1902 1903 1904 1905

Some highlights What works… What hasn’t… • Visual improvements to letter (+6pp) • “Radically Simplified” approach • Improved messaging and sales pitch letters (-10pp) (+5pp) • Envelope teasers (n.s.) • Addition of ‘bureaucratic’ second reminder (+13pp) examination of tone • Compulsion messaging (n.s.) soon • Soft close (+2pp) • Extended deadlines with hard • Personalisation/Postit notes close (n.s) • Handwritten +8pp (but…) • Plain “official” envelopes • Pre-printed +4pp • Image of postit n.s. (-4.5pp vs coloured envelopes)

Current work • Improved sales pitch for the survey & Social norms • Tone of letter content & escalation over multiple contacts • Additional email and SMS reminders • Targeted strategies: Secure apartment buildings, remote populations • Reducing the barriers reported by nonrespondents • Improvements to website landing page & response process • “Retaining” respondents over 8 months of LFS

“Why don’t you use a factorial design? …it’s much more efficient… …and would tell you about the interactions”

Which design? Factorial design RCT program is… • Efficient use of sample • highly responsive, • Estimate interaction terms • maintains ecological and internal validity, • is relatively robust to • Quickly become large and operationalisation errors, complex • High effort & management costs • conducted at very low cost = single, one off experiment • dramatically increasing adoption • Vulnerable to implementation errors

…highly responsive, maintains ecological and internal validity, is relatively robust to operationalisation errors, and yet is Responsive conducted at very low cost while dramatically increasing adoption The monthly, low-effort design: • Increases willingness to test more risky/unconventional approaches • Is capable of informing immediate issues • Easily integrate new ideas from outside the organisation • Adapts to emerging and evolving interests, theory and problems • Frequent results maintain interest in the experimental program and respondent behaviour generally