Science of f Significance How an e-commerce company went from Wild West testing rules to wildly successful results through variance testing Em Emil ily Em Emmer Alli lison Ban Banko Senior Interactive Marketing Manager Reporter Extra Space Storage MECLABS
Session speaker Emil ily Emmer Sen enior In Interactiv ive Marketi ting g Manager Extra Space Storage • At Extra Space Storage, Emmer has increased the conversion rate by 130% over the past four years. • Her group has increased Web performance 1,300% and delivered more than $50 million of incremental revenue annually. @emilyemmer • Extra Space Storage tests and optimizes across channels including desktop, mobile, email, call center and online advertising. • Emmer holds her MBA from W.P. Carey School of Business at Arizona State University and a B.A. from the University of Utah. 2
About Ext xtra Space Storage • Publicly traded company founded in 1977 • Nationwide with 1,100 self-storage properties • 650,000 customers with 500,000 new customers annually • $1 billon in revenue annually
Wild West testing rules
Homepage Contr trol Treatment 15% Lift
A deeper lo look in into the test Control Treatment 475 Conversions 413 Conversions 12% Traffic 12% Traffic 3 Weeks Testing 3 Weeks Testing 15% Lift
Actual dif ifference or varia iance?
What is variance testing? A/A test of f identical experiences. Instead of absolute difference in conversion rate, this test measures the natural variability of a website.
Varia iance vs. . conversion testing A/B Testing A/A Testing Difference (lift) Variance (noise) Results based on Results used to difference between determine testing treatment and rules control 15% Lift
Wildly successful testing
Varia iance testin ing help lped us understand: • How long do we run our tests? • How much traffic do we need for each treatment? • How many treatments do we run in our tests? • How big must the lift be to call a winner? • How many conversions do we need per treatment?
Extr xtra Space Storage varia iance testin ing Loc Location: ExtraSpace.com Over erview: 5 exp xperiments Res esea earch Notes: Bac Background: Company needed testing rules to determine a significant result. Objective: Determine the baseline noise of the website for reading future tests. Ob Prim rimary ry Res esearch Qu Question: What conditions make our tests confident? Tes est t Des Desig ign: Same experiences tested across different amounts of traffic.
Exp xperim iment 1: : 35% tr traffic ic per r tr treatment (2 tr treatments) 1% difference with 3 weeks traffic Relatively small amount of variance with 35% traffic per treatment. Week 2 Week 3 Control 2.03 2.02 Control B (Same Experience) 1.98 2.02 Rel elative Dif Difference 2.08 2.08 1.04 1.04
Exp xperim iment 2: : 18% tr traffi fic per r tr treatment (4 (4 tr treatments) 4% difference with 3 weeks traffic Higher natural variance with 18% traffic per experience. Week 2 Week 3 Control 1.94 1.98 Control B (Same Experience) 2.62 2.05 Rel elative Dif Difference 4.65 4.65 3.67 3.67
Exp xperim iment 3: : 12% tr traffi fic per r tr treatment (6 (6 tr treatments) 14% difference with 3 weeks traffic Significant variance. Week 2 Week 3 Control 1.55 2.02 Control B (Same Experience) 1.91 1.76 Rel elative Dif Difference 23 23.56 .56 14 14.44 .44
Exp xperim iment 4: : 7% tr traffic ic per r tr treatment (6 (6 tr treatments) 20% difference with 4 weeks traffic Substantial noise at this volume of testing. Week 3 Week 4 Control 2.02 1.78 Control B (Same Experience) 1.77 2.13 Rel elative Dif Difference 18 18.70 .70 19 19.69 .69
Exp xperim iment 5: : 3% traffic ic per r treatment (6 treatments) 17% difference with 5 weeks traffic Large difference at this volume of testing traffic. Week 4 Week 5 Control 2.03 2.17 Control B (Same Experience) 1.77 1.85 Rel elative Dif Difference 15 15.11 .11 16 16.89 .89
Summary: Varia iance by y traffic ic and tim ime perio iod Traffic Time A Time B Experiment 1 35% 2% 1% Experiment 2 18% 5% 4% Experiment 3 12% 24% 14% Experiment 4 7% 19% 20% Experiment 5 3% 15% 17%
Our testing rules • Need to test at least 15% of volume for A/B tests • Lift needs to be more than 4% to be called a winner • Need 500 conversions per experience • Run tests for at least two weeks
Our testing rules • Need to test at least 15% of volume for A/B tests 45% increase in conversion rates • Lift needs to be over 4% to be called a winner • Need 500 conversions per experience • Run tests for at least two weeks
How can I I start?
When should I conduct varia iance testin ing? • At different times of the year if your website has seasonal differences (i.e., high/low season variance) • Repeat variance testing if you notice dramatic changes to your traffic • Repeat variance testing after any site changes are made that affect visitor behavior 27
5 Sim imple le Tests: C Complete these varia iance tests sim imultaneously to develop testing rule les for your website Test Test Test Test Test 50% Traffic 33% Traffic 25% Traffic 20% Traffic 10% Traffic 2 Experiences 3 Experiences 4 Experiences 5 Experiences 10 Experiences
Analy lyze Results: Compare result lts wit ithin in each test and across tests Absolute Variance Conv. Experience Visitor Conversions Rate Vs. A Vs. B Vs. C Vs. D A - Default 24,241 482 1.99% -- 1.98% 2.55% 2.53% B - Default 24,208 472 1.95% 1.94% -- 4.45% 4.43% C - Default 24,210 494 2.04% 2.62% 4.65% -- 0.02% D - Default 24,264 495 2.04% 2.60% 4.63% 0.02% --
Alt lternatively: In Integrate varia iance in into testing A/B Test A/A/B Test Control Control Treatment Control (V2) Treatment Treatment
Key takeaways • Variance testing is EASY and has long-term benefits • Just test setup is required, no develo lopment • Gain consensus about test setup, duration and results • Get guidance for significant results through testing
Thank you Emil ily Emmer Senior Interactive Marketing Manager Extra Space Storage @em emil ilyemmer Allis llison Ba Banko Reporter MECLABS @alli allisonbanko
Recommend
More recommend
