Case Study: How Consumer Source, Inc. uses multivariate testing to make intelligent business decisions and ramp up conversions Paul Terry Sr. Manager - Optimization Consumer Source – A Primedia Company
Case Study Background
Case Study Background • Who we are: Consumer Source, a Primedia company Business established in 1975 with the print publication Apartment Guide – a free directory of apartment communities. • Primary Business: Lead generation for subscription based advertisers on the internet and in print 3
Case Study Background • ApartmentGuide.com, Rentals.com, NewHomeGuide.com, AmericanHomeGuides.com, RentalHouses.com, Rentlist.com • Advertisers increasingly want more internet presence • Early site improvements were HIPPO driven • Development cycles were long • Analytics were fuzzy • Results were obscure 4
Case Study Background • In 2008, incoming executive management began to transform Consumer Source into an internet company. • Efforts & budgets for SEO and SEM stepped up • Agile development methodology adopted • A Website Optimization team was created, and SiteSpect was selected to implement multivariate testing 5
Case Study Background Most testing involves no involvement • from IT; some graphic design Testing objectives typically to increase • lead conversion and advertising revenue Run A/B or multivariate tests on several • websites; 5-6 concurrent per site. Winners become Agile stories for • implementation Some winners are rolled out to other sites • 6
Tactics: Test (nearly) Everything
Tactics: Test (nearly) Everything 8
Tactics: Test (nearly) Everything Tactic #1: Hone Calls to Action Original : “Contact Property” Users prefer : Why do I want to do that? Check Availability Current Rent Specials 9
Tactics: Test (nearly) Everything Tactic #2: Improve the location of the Lead Form Expert Opinion : Main photo should be “front and center” Users prefer : Make it obvious how to proceed. 10
Tactics: Test (nearly) Everything Tactic #3: Improve the lead form Original : First and Last name Users prefer : One name field 11
Tactics: Test (nearly) Everything Tactic #3: Improve the lead form Require Phone Number Some Clients prefer us to require the phone number 12
Tactics: Test (nearly) Everything Tactic #3: Improve the lead form Original : Move-in Date Calendar control Users prefer : Immediately, Next week, Next month, etc. 13
Tactics: Test (nearly) Everything Tactic #3: Improve the lead form Original : Small black “ Submit ” button Users prefer : Large red “ Send” button. 14
Tactics: Test (nearly) Everything Tactic #4: Search Header 15
Tactics: Test (nearly) Everything Tactic #4: Search Header Original : Page headers varied Users prefer : Consistent treatment promotes ease of use 16
Tactics: Test (nearly) Everything Tactic #5: Search Type Ahead Running series of tests to optimize how the type ahead functions 17
The Challenge – Increase value to ALL advertisers
Increase value to ALL advertisers MVT Lab Challenge: Create a tide to lift all boats! fold 19
Increase value to ALL advertisers Goal #1: Increase leads to primary advertisers Important! fold 20
Increase value to ALL advertisers Goal #2: Maintain or I ncrease clicks to secondary advertisers Also Important! fold Express value of Content A in terms of Content B 21
Increase value to ALL advertisers Goal #3: Draw users below the fold Small intra document links Rich content shouldn’t fold be missed 22
Increase value to ALL advertisers Opportunity : Photos are hidden fold 23
Increase value to ALL advertisers Four main designs • Three multivariate and 8 A/B tests • Twenty Variations • 24
Increase value to ALL advertisers Four main designs • Three multivariate and 8 A/B tests • Twenty Variations • Conventional wisdom challenged • 25
Increase value to ALL advertisers Four main designs • Three multivariate and 8 A/B tests • Twenty Variations • Conventional wisdom challenged • Three quarters of a million tested users selected the • winning solution for our advertisers 26
Increase value to ALL advertisers fold 27
Increase value to ALL advertisers Community Carousel Auto advance slideshow • Mouse wheel scrolling • 10 initial photos shown • Ajax driven • Click to enlarge photo • Click to load all photos • fold 28
Increase value to ALL advertisers - Results Community Carousel 21% more visitors send • a lead 26% more leads per visit • fold 29
Increase value to ALL advertisers - Results Community Carousel 21% more visitors send • a lead 26% more leads per visit • 15% more Credit Score • clicks fold 30
Increase value to ALL advertisers - Results Community Carousel 21% more visitors send • a lead 26% more leads per visit • 15% more Credit Score • clicks fold 45% more users scroll to • the bottom of the page. 31
Test Analytics
Test Analytics Time – Identifying normal cycles helps you to know how long to run a test The final Community Carousel test iteration converging to a 26% increase in leads was verified when the feature went to production 33
Test Analytics Time – Identifying normal cycles helps you to know how long to run a test Segmentation –Understand and eliminate outlying data. Specific browsers, succeeding users, failing users Community Carousel was even more effective on IE8, our highest converting browser. Lead conversion +35% 34
Test Analytics Time – Identifying normal cycles helps you to know how long to run a test Segmentation –Understand and eliminate outlying data. Specific browsers, succeeding users, failing users Confidence – Statistical calculation “Two Tailed Test” expresses statistical significance of results Confidence ratings above 90% indicate high statistical significance. 35
Test Analytics – The Story Every user tells a story Define a common set of metrics across all tests See how your test changes the story for each segment The Community Carousel caused the nearby lead call to action to more than triple its conversion 36
Summary & Key Takeaways
Summary & Key Takeaways Test the small things - Page layout, positioning and simplicity of forms, colors, breaking below the fold Hone calls to action – Know what your users want, and discover how best to lead them there Test the big things - Bold features and new designs can be fleshed out, refined, and put into production with confidence Measure competing content in terms of each other Establish a framework of common metrics Testing practice should work well with development process – Agile! There are NO failures. Always gain better understanding of your site and users 38
Question-and-Answer Session 39
Recommend
More recommend
