Democratizing online controlled experiments at Booking.com How to run 1.000+ concurrent experiments Lukas Vermeer | Director of Experimentation
Booking. Founded in 1996 28,000,000+ reported listings 1,550,000+ room nights per day 17.000+ employees 1.000+ concurrent tests 2.000+ people testing
TL;DR. Central repository of successes and failures Descriptions of all experiment iterations and of the final decision are available for all experimenters. Genericity and extensibility Experimental design is abstracted away. Reports are automated and product agnostic. Data which can be trusted We monitor the validity of the data by computing common metrics in two separate date pipelines. Loose coupling Business logic and experiment infrastructure are purposefully kept decoupled. Building safeguards We encourage sound methodology and provide data quality checks, but no rules or automation.
Focus.
This is a story about culture
This is a story about people
Method. 1. Ask PR department for “ pretty photos of Sample employees ”. 2. Filter out non-experiment roles (dba, support, management, etc). 3. Select six at random from the remainder. Survey 4. Email them and ask “ how do you use experiments in your job ”. 5. Copy paste replies onto slides. Results 6. Highlight key phrases . 7. Sprinkle in some context.
Empower people to experience the world.
“Data-driven is quite common these days in tech industry, however, empowering everyone to make data-driven decisions independently is quite unique in Booking.com.” Clyde Li. Client Side Developer
“I can come up with an idea over breakfast, bike to the office and have it live well before lunch. I’ve never worked anywhere else that gives me this level of ownership and creative freedom to validate my ideas. ” Nekeia Boone. Senior UX Copywriter
“I think it's great that the company encourages everyone to hack and test ideas among the users. No higher value opinions, no centralization of decision making , everyone is free to have their own ideas and at the end of the day, users are the ones who decide what's best for them.” Heloisa Biagi. Client Side Developer
You cannot have a democracy without empowering the people
“Experimentation is a great way to figure out if something as small as a copy change or something as big as a whole book process flow will help users, but you can’t depend on it as your only source of product validation as this will greatly hinder the product development.” Hadeer Younis. Full-Stack Software Developer
“The plural of anecdote is not data” - Lots of misinformed speakers at conferences I’ve attended
“The plural of anecdote is not data” - Raymond Wolfinger (1969)
50% 60% 70% 80% Data is just data. To make good decisions, we need good evidence . White Blue Orange Grey
Website Optimisation. Let’s change the button from yellow to blue and see if it increases the magic number. A B Buy now! Buy now!
Hypothesis Testing. We observed in user research that some people have difficulty finding the “buy now” button. We suspect this is caused by the low contrast between the font and the background. To solve this user issue, we will change the button from yellow to blue . If this solution works, we expect to see more users hover and click, and eventually purchase. A B Buy now! Buy now!
“We use experimentation to help us validate hypotheses with the goal of addressing well defined user problems . It's all about learning as fast as possible in the most rigorous way possible.” Finn Hansen. Product Owner
Hypothesis template. Theory Based on [prior] we believe [condition] for [users] will encourage them to [behavior] Validation We will know this when we see [effects] happen to [metrics] Objective This will be good for customers, partners and our business because [motivation] .
10% 9/10 tests “fail”. “90% of product decisions have an inconclusive or negative effect on a product’s primary metric.” - VWO (2016) 90%
https://medium.com/booking-com-development/moving-fast-breaking-things-and-fixing-them-as-quickly-as-possible-a6c16c5a1185
https://medium.com/booking-com-development/moving-fast-breaking-things-and-fixing-them-as-quickly-as-possible-a6c16c5a1185
“We work in a global scale. By validating my changes through experimentation I continuously get challenged on my assumptions and the way I look into the experience of our customers.” Diogo Antunes. Principal Developer & Fellow
Disclaimer: not what the actual experiment looked like.
Assumptions. Order The order of items can be improved . The order of items matters to users . Feature The feature can be improved . The feature matters to users . Page The page can be improved . What is on the page matters to users .
Take the biggest small step so you can challenge your riskiest assumptions quickly
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
http://lukasvermeer.nl/confidence
Thank you! www.workingatbooking.com
Recommend
More recommend
