Testing Email Invitations in Testing Email Invitations in a - PowerPoint PPT Presentation

Testing Email Invitations in Testing Email Invitations in a Nonprobability Panel p y Elizabeth Nichols, Ryan King and Elizabeth Nichols, Ryan King and Jennifer Childs U S C U.S. Census Bureau B This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U S Census Bureau The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.

Why is the U.S. Census Bureau interested in researching email notifications?

Motivation for nonprobability split-panel testing  Quantitative data: Quantitative data:  Too many choices to test in large mandatory tests  Qualitative data:  Does opinion data match real response data? p p

Previous research Previous research  Subject line S bj li  Response deadline (Henderson, 2011 AAPOR)  Topic or no topic (Couper, 2008)  Avoid “survey” (Porter & Whitcomb, 2005)  Shorter the better & Avoid “Help” & “Reminder” (Genroe – Marketing)  Email content (Klofstad, Boulianne, Basson, 2008)  Informing about reminders  Longer email content

Six studies in 2014 Six studies in 2014 Jan. March h May Aug. Oct. Dec. Subject x x x line line Email x x format Email x x content Survey Survey x link type Due date x Sent x x time

Nonprobability sample Nonprobability sample  Highly educated  >35 yrs 35 old  White  White  Employed   Census Census

Response rate (RR2) Response rate (RR2) 25% 21% 20% 20% 14% 15% 13% 13% January (n=1000) ( ) March (n=1500) 10% May (n=1500) y ( ) 5% 0% Overall

Subject lines Subject lines  January J  Confidential 2014 Census Study  Respond to the 2014 Census Study Respond to the 2014 Census Study  March  Confidential 2014 Census Study y  Help us make the U.S. Census better, answer our survey  A message from the Director of the U.S. Census Bureau  May  Confidential 2014 Census Survey  10 min te U S Cens s S r e to Help o r Comm nit  10-minute U.S. Census Survey to Help your Community  U.S. Census Survey to Help your Community

Motivation for January Research team decision  Confidential 2014 Census Study  Short  Emphasizes confidentiality  Respond to the 2014 Census Study R d t th 2014 C St d  Short  Call to action

January open rate January open rate 40% 40% 35% 30% 30% 25% 20% 0% Confidential fid i l 15% Respond 10% 5% 0% Source: U.S. Census Bureau January Small-scale Nonprobability Panel Research y p y

January response rate January response rate 25% 21% 20% 20% 15% 10% 5% 0% Confidential Respond Source: U.S. Census Bureau January Small-scale Nonprobability Panel Research

Survey feedback from January Survey feedback from January

Results of opinion question Results of opinion question 38.4% Help us make the U.S. Census better, answer our survey 32.0% 10 minutes to improve the U.S. Census and help your community 31.6% Answer the U.S. Census– Help your community 30.1% Important Census Study 26 7% Improve the U S Census Study 26.7% Improve the U.S. Census Study 13.6% Mandatory Census Study 13.1% Your Civic Duty – Answer the U.S. Census! 6.8% A message from John Thompson, Director of the U.S. Census Bureau 6.3% A message from the Director of the U.S. Census Bureau

Motivation for March Motivation for March  Confidential 2014 Census Study  Kept January “winner” p y  Help us make the U.S. Census better, answer our survey our survey  Winner of January opinion question  A message from the Director of the U.S. Census Bureau Census Bureau  Loser of January opinion question

March open rate March open rate 45% 45% 40% 35% 30% 25% Confidential 20% 20% Help 15% A message 10% 5% 0% Source: U.S. Census Bureau March Small-scale Nonprobability Panel Research Source: U.S. Census Bureau March Small scale Nonprobability Panel Research

March response rate March response rate 25% 20% 17% 15% 15% 11% 11% 10% 5% 0% 0% Confidential Help A message Source: U.S. Census Bureau March Small-scale Nonprobability Panel Research

Motivation for May Motivation for May  Confidential 2014 Census Survey f d l  Kept with the March winner  10-minute U.S. Census Survey to Help your Community  Email provider suggestion that numbers do well  Emphasize U S Census  Emphasize U.S. Census  Data use: Helping a community  U.S. Census Survey to Help your Community U S C S t H l C it  Like previous panel, but no number

May open rate May open rate 45% 40% 35% 35% 30% 25% Confidential Confidential 20% 10-min 15% Help 10% 10% 5% 0% 12-May 13-May 14-May 15-May 16-May 17-May 18-May 19-May 20-May 21-May 22-May 23-May 24-May 25-May 26-May 27-May Source: U.S. Census Bureau May Small-scale Nonprobability Panel Research

May response rate May response rate 25% 20% 15% 15% 12% 12% 11% 10% 5% 0% Confidential 10-minute Help Source: U.S. Census Bureau May Small-scale Nonprobability Panel Research

Take aways Take aways  Subject lines  Numbers resonate  Confidential seems to do okay  Opinion data does not match actual response data  Opinion data does not match actual response data  “Help” might not help response when it is the first word d  We did not see evidence that shorter is better, but the first words seem to matter

Future plans Future plans  Replicate with probability frames  Administrative record frame of emails  Replicate with other nonprobability frames

Contact Contact Elizabeth Nichols elizabeth.may.nichols@census.gov elizabeth.may.nichols@census.gov