The Biggest Optimization Mistake I Ever Made Case Study: Air Canada Tim Burgess Justin Bridegan Web Analytics Manager Senior Marketing Manager Air Canada MECLABS
Presenters Tim Burgess Web Analytics Manager Air Canada Justin Bridegan Senior Marketing Manager MECLABS 2
Where I Came F rom… Agile Selling primarily to new visitors Experimental with both software and marketing Failure was accepted as part of the learning process Don’t ask about the product quality 3
Testing at Air Canada Unknown or brand new to most individuals and groups Measurement is done through old database architecture Company structure is separated into silos Web analytics and optimization are positioned in IT 4
Web Analytics Maturity Model Mar 2012 Jan 2011 Tools Management 2 Tools Management 2 1 1 Methodology Methodology Objectives Objectives 1 1.5 2 0 Resources Scope Resources Scope 1 1 2 3 ***This has been stolen from Stephan Hammel 5
6
Before I tell you about the worst test I ever had…. …don’t feel bad for me… • We are installing an enterprise-level testing tool • We are gaining ground in the marketing community • My boss still likes me (as far as I know) 7
5 steps for developing a test within Air Canada 1. Develop a strategy – where is the lowest-hanging fruit for profitability? 1 2. Use the MarketingExperiments concepts of Friction and Value to develop mockups 2 3. Find a stakeholder whose interest I will work in, and sell them on the concept 3 4. 4 Set up and run the test 5 5. Report to stakeholders 8
Step 1: Strategy Develop a strategy – Where is the lowest-hanging fruit 1 for profitability? • Test to improve seat selection – Air Canada’s largest source of ancillary revenue 9
Step 2: Analyze Friction and Value Use the MarketingExperiments concepts of Friction and 2 Value to develop mockups W HY DO PEOPLE SAY “ YES ”? Perceived Cost Perceived Value BPY=Pc(VF>CF) BPY=Pc[Cl+Cr](VF[Ap+Ex]>CF[Rc+Ef]) 10
The Original: Seat selection The original variation is good from a navigational point of view, but does nothing to emphasize the value of “Advanced” or “Preferred” seat selection. 11
Variation A This variation focused on bringing Value to the top of the page through bullets, while maintaining clear navigation. 12
Variation B The goal of Variation B was to bring navigation further above the fold, while still featuring the bullet points. 13
Variation C Answered the question, “What if they really don’t know the difference between Preferred and Advanced seat selection?” by promoting description in text form above navigation. 14
Step 3: Find a stakeholder Find a stakeholder whose interest I will work in, and sell 3 them on the concept 15
Stakeholder – Revenue Management The group that manages pricing, loads and network and also responsible for Seat Selection revenue. Loved the idea! 16
Step 4: Set up and run the test Control 4 Testing Variable A Variable B Variable C 17
Step 5: Report to stakeholders 5 Report on findings We ran the test for one week to 95% significance, with a follow-up test for three weeks, that showed that variation C was generating 48.78% more revenue than the original. Variable C Control 18
We have a winner… Variable C So we celebrated! 19
One small hitch … The numbers in the months following the experiment did not reflect the experiment results My Mistake: By communicating the results the way I did, I led the stakeholders to believe that we would see those numbers once the test went live. • Revenue Management did not understand how this could happen • My boss was disappointed • I was confused 20
So, what happened? • Visitors on Aircanada.com are often repeat visitors • There are many points in the process where visitors can purchase seat selection (see the email below) • We cannibalized on other channels 21
Analytics data Even though analytics data showed a change in the amount of revenue for Preferred Seat selection, Revenue Management was used to looking at its own data, which is not slice and trend-able like analytics data. Using conservative numbers the actual impact of the test was an approximate increase of 5% on Preferred Seat selection revenue. Preferred Seat Revenue/flight order Advanced Seat Selection Revenue/Order 7 4 6 3.5 5 3 4 2.5 3 2 1.5 2 1 1 0.5 0 0 Jul 1, 2011 Jul 6, 2011 Jul 11, 2011 Jul 16, 2011 Jul 21, 2011 Jul 26, 2011 Jul 31, 2011 Aug 5, 2011 Aug 10, 2011 Aug 15, 2011 Aug 20, 2011 Aug 25, 2011 Aug 30, 2011 Jul 1, 2011 Jul 6, 2011 Jul 11, 2011 Jul 16, 2011 Jul 21, 2011 Jul 26, 2011 Jul 31, 2011 Aug 5, 2011 Aug 10, 2011 Aug 15, 2011 Aug 20, 2011 Aug 25, 2011 Aug 30, 2011 22
The irony… …is that this is the most successful test I have run to date, in terms of profitability. 23
Key takeaways Remember the big picture Manage expectations Predict scenarios, not results Keep testing Remember… The way in which hypotheses and results are presented is pivotal to building support and making optimization a long-term strategy. 24
Thank you! Questions? 25
Recommend
More recommend
