Building a culture of data-informed decision making: lessons in one year of data analytics at Slack Sarah Edge Mann, PhD April 24, 2017
Me & Analytics at Slack
University of Arizona (Aug 2008 - Aug 2013) ● PhD in Applied Mathematics ● Minor in Computer Science ● Dissertation work in informations theory: ○ Error correcting codes ○ Fast decoding algorithms
Facebook: Data Scientist (Oct 2013 - Feb 2016) Worked on: ● Growth ● Interfaces for low end phones, slow internet connections ● Facebook Lite
Slack: Growth Analytics Lead (Feb 2016 - Present)
Slack Data Analytics Mission: Help the company make data informed decisions ● What product areas should we invest in? ● What features should we ship? ● How do people use our product? ● What is the health of our business? ● What drives our growth? ● What can we do to drive our growth? ● How can we spend money to acquire users? ● What are our capacity needs in the next year?
Slack Data Analytics What we do Behind the scenes: ● Ensure high quality logging - you can’t know what you don’t log! ● Curate high quality data sets, make it easy to answer common questions ● Build tools to make data accessible to everyone at Slack - data to the people! With the product and business teams: ● Goal setting ● Goal tracking ● Understand long term trends and usage patterns ● Scope reach and impact of potential projects ● Prioritization ● Support A/B testing, should we launch a feature?
Slack Data Analytics What we don’t do ● Machine Learning ● Artificial Intelligence ● Build user facing products Slack’s Search, Learning and Intelligence team does these things.
When should you build an analytics team? ● You have achieved basic product market fit ● You have a large user base - analysts need large data sets to be useful
You need to have data to analyze it Build a data engineering team before building an analytics team. Data Engineering: data availability Data Analytics: voice of data within the company
Slack’s Data Team Data Analytics: 20 people Data Engineering: 10 people Product: 6 Infrastructure: 4 Growth & Marketing: 7 Product: 5 Business & Sales: 3 Analytics Tools: 3
Feature development Two Case Studies
Team communication for the 21st century.
Case Study: Invites in the team creation flow
Case Study: Invites in the team creation flow Invites removed from team creation flow Health of new teams Christmas holidays Invites added back into Ad campaigns and PR team creation flow events Date of team creation
Case Study: Invites in the team creation flow Lesson learned: You don’t know what you don’t track!
Case Study: Mobile team creation Slice metrics to find gaps Teams created on desktop Health of new teams Teams created on mobile Date of team creation
Case Study: Mobile team creation Dig deeper to identify opportunities % of new teams that send invites Teams created on desktop Teams created on mobile Product work on mobile Date of team creation
Tools & Process Keys to a data driven culture
Slack has an Analytics Tools Team who develop: Data visualization tools Experiments framework
Data visualization tools ● Democratization of data: enable PMs & engineers to access data ● Minimize the time between data questions and answers
Experiments Framework ● Science! ● Measure the impact of every change you make
Example A/B test Control Test Get Started
Experiment Framework: Diversions Who should see which experience? ? Get Started
Experiment Framework: Exposure Logs Who saw which experience? Get Started
Experiment Framework: Metrics, logging What did people do? Click Click Get Started
Experiment Framework: Compute, Visualize Results Which experience performed better? 4% Click Through Rate Click Through Rate: +25% 95% Confidence Interval: 18% - 32% 5% Click Through Rate Get Started
That sounds complicated….
So you also need process
We do two things in Growth to help make experiments run smoothly: 1) Weekly experiment reviews 2) Bi-weekly numbers review
Experiment Reviews ● Meet weekly in small groups ● Everyone involved with the feature should attend: ○ Product manager, engineers, design, analyst, QA ● Check in on all experiments: ○ During development ○ Shortly after experiment launch ○ Upon conclusion ● This is a highly participatory meeting!
Growth Numbers Review ● Held bi-weekly, includes the entire growth team ● Product managers present a synthesized story about each feature ○ What was tested and why ○ How did they test it ○ Experiment outcome ○ Lessons learned, suggestions for future work ● Deck gets shared company wide + a short synopsis of highlights
Benefits of this process ● Data literacy ● Quality ● Speed ● Accountability ● Visibility ● Professional development
Case Study: Allow bulk inviting in the team creation flow. Eligibility Any team with an email domain identified to be using GSuite (via MX record). Hypothesis Giving team creators that use GSuite an option to import address books will improve invite sends / sender and team activation.
In V1.0, we saw some concerning declines. ● The number of teams reaching 3+ users was down 5% ● Users joining teams from a shared invite link was down dramatically
We made two small changes - V1.1 Many users were clicking “send” with no recipients selected. So we grayed out the Send Invites button unless it would actually send. ● Saw lower adoption and usage of invite links. So, we added a link to the UI. ● Results Invites sent up 7% ● All metrics neutral to positive ●
Goals
Goals: What they are ● A metric + target value ● A statement about priorities You should typically have company level goals as well as team level goals.
Goals: Why set them? ● Connects feature work to the company’s mission and success ● Creates accountability ● Forces eyes on the numbers Goal Metric ● Pushes you to understand your funnel deeply Date
Process Around Goals ● Dashboards that track progress against goals ● Weekly written reports on our goals ● Start our Numbers Review Meeting with an update on goals ● All growth projects tie back to one of our goals
Growth Analytics On Call Rotation
On Call: What we do One growth analyst is on call each week. They are responsible for: ● Triaging data availability and quality problems ● Answering ad hoc questions about growth data ● Writing our weekly reports on growth metrics and goals
On Call: Benefits A better team! ● Shared responsibility for our data ● Highly collaborative team ● Better code reviews ● No one person is a single point of failure ● Everyone on the team understands all of our growth metrics ● Space to focus when you aren’t on call
Questions?
Recommend
More recommend
