spotify lessons learning to let go of machines
play

Spotify Lessons: Learning to Let Go of Machines James Wen, Site - PowerPoint PPT Presentation

Dec 22, 2023 •256 likes •938 views

Spotify Lessons: Learning to Let Go of Machines James Wen, Site Reliability Engineer at Spotify ALF Squad, Infrastructure & Operations Tribe IO Tribe Lets control how feature developers think about what their code is actually

  1. Spotify Lessons: 
 Learning to Let Go of Machines James Wen, Site Reliability Engineer at Spotify 
 ALF Squad, Infrastructure & Operations Tribe IO Tribe

  3. Let’s control how feature developers think about what their code is actually running on.

  4. Takeaways • Feature developers = happiest with feature work • Find out developer machine concerns and mitigate • Migrating to cloud or hybrid? Start embracing ephemeral service design and infrastructure

  5. Agenda • Why? • Journey • Hybrid Cloud • Ops in Squads • Future • Learnings

  6. Why? Why don’t we want feature devs to care too much about infrastructure and machines?

  7. Why? Time taken on infrastructure tasks = time taken away from feature work Feature devs = focused on features

  8. Spotify Scale Stats - 140 Million+ Monthly Active Users - 50 Million+ Subscribers - 30 Million+ Songs - 2 Billion+ Playlists - Available in 60 markets

  9. Spotify Dev Scale Stats ~900 Devs ~100 Tech Teams ~2000 Services

  10. Spotify Machine Scale Stats ~10,000 Bare Metal Hosts 
 ~13,000 Hosts on GCP 
 46 Hardware/VM Types

  11. Example: Capacity Planning Avg # devs on a team Capacity Planning

  12. Scale doesn’t really matter -Smaller companies/teams = developer time is more valuable -Larger companies/teams = wasted infra time scales as well

  13. Other Infrastructure Tasks - Machine provisioning 
 - Failure planning - Security updates - Machine maintenance

  14. Dedicated Ops?

  15. Dedicated Ops? ~2000 Services 
 74 Infrastructure and Operations Engineers If all IO engineers → dedicated ops 
 27:1 service:engineer ratio

  16. 
 Ops In Squads Feature teams handle their own ops and provisioning 
 Using the services and tooling the Infrastructure and Operations tribe has written

  17. We control the level of context feature teams need to operate their services.

  18. - Developer Happiness 
 - Developer effectiveness and context

  19. Journey

  20. - Ops in Squads - Hybrid Cloud (Ephemerality)

  21. Starting Out

  22. Historical: Feature Developer’s Context for Service’s Capacity San Jose Stockholm Rack 1 Rack 2 Rack 2 lon-1-a lon-1-c lon-1-e keys lon-1-b lon-1-d lon-1-f updated updated

  23. Machine Context - Packages Unbound 
 v1.6.3 - Hostname - Machine specs (CPU, RAM, disk, etc.) Openssl v1.0.0f - Uptime and service duration - Location - Local state (files on disk, info in 8 GB 2 Cores 3 Years RAM memory) In Virginia Tarred Logs ash2-metadata-a.ash2.spotify.net

  24. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? Specs? How many?

  26. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? Specs? How many?

  27. ServerDB

  28. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  29. ProvGun/ProvCannon

  30. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  31. DNS

  32. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  33. Nameless

  34. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  35. Cortana

  36. Cortana

  37. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  38. Helios and Containers

  39. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  40. Google Compute Platform

  41. ash2-cortana-a1.ash2 
 Zone Service Group Sequential # gew1-cortana-a-l33t.gew1 Zone Service Pool Random 4 Chars

  42. Cortana Pool Manager

  43. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  44. Regional Managed Instance Groups

  45. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Up to date? How long? Where? Available? How many? Specs?

  46. MBMI: Minimal Base Machine Image

  47. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  48. Phoenix

  49. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  50. Current: Feature Developer’s Context for Service’s Capacity Stockholm GCP - europe-west-1 Pool: 
 Pool: 
 4 instances x (High Mem) 2 instances x (n1-standard-32)

  51. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  52. Future

  53. Gordon (Cloud DNS)

  54. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  55. Autoscaling

  56. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  57. Right Sizing

  58. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  59. Future Feature Developer’s Context for Service’s Capacity GCP - asia-east-1 GCP - europe-west-1 Service Pool Service Pool GCP - us-central-1 Service Pool

  60. Feature Developer Concerns How to How to talk How to get? Service + Business track? to it? What tools What to put Maintenance? on it? on it? Where? Up to date? How long? Available? How many? Specs?

  61. Learnings

  62. Why Pets to Cattle was Difficult: - Manual/tedious setup 
 - Wait times for machine becoming ready (packages, DNS) 
 - Non-automatic security updates - A fixed, reliable hostname - SSH Access - Always up/present unless team tears down

  63. Ephemerality Learnings - Monitoring 
 - Logging 
 - Service Design - Incidents

  64. Hybrid Learnings - Replicate bare metal functionality, then iterate - When in doubt, devs provision up and many - Migration = great time to influence dev paradigms - Don’t need to DIY

  65. 
 DevEx Learnings - Feature devs need carrots, sledgehammers, and/or limos to change - Edge Cases: REST API + CLI = provide enough for feature teams to handle the edge cases

  66. Recap - Decrease necessary infrastructure context - Increase reliability - Save $$$ - Increase dev happiness and productivity

  67. Let’s strategically control and limit how feature developers think about infrastructure.

  68. 
 James Wen 
 Email: jameswen@spotify.com 
 Twitter/Github: @rochesterinnyc LinkedIn: jamesrwen 
 Spotify is hiring! spotifyjobs.com IO Tribe

Recommend

Scaling Data Infrastructure @ Spotify matti@spotify.com kalvans@spotify.com Mrti Kalvns

Scaling Data Infrastructure @ Spotify matti@spotify.com kalvans@spotify.com Mrti Kalvns

Scaling Data Infrastructure @ Spotify matti@spotify.com kalvans@spotify.com Mrti Kalvns Matti Pehrs kalvans@spotify.com matti@spotify.com Agenda 1. Data at Spotify 2. Summer of 2015 3. Challenges & Victory Datamon

543 views • 42 slides
The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com)

The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com)

The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com) @l_phant @ravwojdyla Technical Product Owner Data Engineer Data Infrastructure Hadoop Team Overview Growing Pains Gaining Focus The

945 views • 62 slides
The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com)

The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com)

The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com) @l_phant @ravwojdyla Technical Product Owner Data Engineer Hadoop Squad Hadoop Squad Overview Growing Pains Gaining Focus The

774 views • 51 slides
Big Data at Spotify Anders Arpteg, Ph D Analytics Machine Learning, Spotify Quickly about me

Big Data at Spotify Anders Arpteg, Ph D Analytics Machine Learning, Spotify Quickly about me

Big Data at Spotify Anders Arpteg, Ph D Analytics Machine Learning, Spotify Quickly about me Quickly about Spotify What is all the data used for? Quickly about Spark Hadoop MR vs Spark Need for (distributed)

357 views • 34 slides
The Spotify Platform WOW Hack Gteborg 2014 Per-Olov Jernberg @possan @SpotifyPlatform Spotify

The Spotify Platform WOW Hack Gteborg 2014 Per-Olov Jernberg @possan @SpotifyPlatform Spotify

The Spotify Platform WOW Hack Gteborg 2014 Per-Olov Jernberg @possan @SpotifyPlatform Spotify APIs, SDKs and Widgets Spotify Widgets Play Button Follow Button Libspotify Embed Spotify in your application or hardware. iOS SDK Beta

326 views • 21 slides
Danielle de Ferrari Sarah de Ferrari Source: Spotify Source: Spotify, 2014 Source: Mashable,

Danielle de Ferrari Sarah de Ferrari Source: Spotify Source: Spotify, 2014 Source: Mashable,

Bryce Simantel Christine Osazuwa Danielle de Ferrari Sarah de Ferrari Source: Spotify Source: Spotify, 2014 Source: Mashable, 2013 Source: Spotify, 2013 WEAKNESSES STRENGTHS High frequency in audio & visual Supports the

921 views • 27 slides
Music Recommendation in Spotify Boxun Zhang About me Data scientist at Spotify Big hype

Music Recommendation in Spotify Boxun Zhang About me Data scientist at Spotify Big hype

Music Recommendation in Spotify Boxun Zhang About me Data scientist at Spotify Big hype nowadays Build models of user behavior Develop algorithms Design A/B tests Ph.D. in CS from TU Delft (NL) Studied user behavior in

799 views • 20 slides
Breaking the hierarchy How Spotify enables engineer decision making Kristian Lindwall, Spotify

Breaking the hierarchy How Spotify enables engineer decision making Kristian Lindwall, Spotify

Breaking the hierarchy How Spotify enables engineer decision making Kristian Lindwall, Spotify Introduction 3 Section name Our employees are our most valued resource So where does it go wrong? 6 Section name Lets do this! 7

571 views • 44 slides
Optimization for Machine Learning Tom Schaul schaul@cims.nyu.edu Recap: Learning Machines

Optimization for Machine Learning Tom Schaul schaul@cims.nyu.edu Recap: Learning Machines

Optimization for Machine Learning Tom Schaul schaul@cims.nyu.edu Recap: Learning Machines Learning machines (Neural Networks, etc.) Forward passes produce a function of input Trainable parameters (aka weights, biases, etc.)

815 views • 50 slides
TICKETMASTER SPOTIFY We are proposing a new way for music fans to purchase concert tickets by

TICKETMASTER SPOTIFY We are proposing a new way for music fans to purchase concert tickets by

TICKETMASTER SPOTIFY We are proposing a new way for music fans to purchase concert tickets by identifying legitimate fans through Spotify, as a means to counter bots and scalpers. Business Problem Among fans and artists, Ticketmasters

539 views • 17 slides
Support Vector Machines 290N, 2014 Support Vector Machines (SVM) Supervised learning

Support Vector Machines 290N, 2014 Support Vector Machines (SVM) Supervised learning

Support Vector Machines 290N, 2014 Support Vector Machines (SVM) Supervised learning methods for classification and regression they can represent non-linear functions and they have an efficient training algorithm derived

1.15k views • 57 slides
Thesis: We will never really understand learning until we build machines that learn many

Thesis: We will never really understand learning until we build machines that learn many

Never Ending Language Learning Tom M. Mitchell Machine Learning Department Carnegie Mellon University Thesis: We will never really understand learning until we build machines that learn many different things, from years of diverse

427 views • 41 slides
Ad Serving at Spotify Scale A journey of incremental full stack overhaul Kinshuk Mishra, Director

Ad Serving at Spotify Scale A journey of incremental full stack overhaul Kinshuk Mishra, Director

Ad Serving at Spotify Scale A journey of incremental full stack overhaul Kinshuk Mishra, Director of Engineering kinshuk@spotify.com @_kinshukmishra A lucky mistake Expected consequences Sarcastic empathy Some valuable feedback The

1.06k views • 88 slides
Building Personable Machines (Lessons from Infusing Affect in Physical Robots) Star Simpson Bot

Building Personable Machines (Lessons from Infusing Affect in Physical Robots) Star Simpson Bot

Building Personable Machines (Lessons from Infusing Affect in Physical Robots) Star Simpson Bot Summit 2016 Victoria & Albert Museum @starsandrobots http://robotic.media.mit.edu/portfolio/tikl/

355 views • 17 slides
Introduction to Support Vector Machines Andreas Maletti Technische Universitt Dresden

Introduction to Support Vector Machines Andreas Maletti Technische Universitt Dresden

Introduction to Support Vector Machines Andreas Maletti Technische Universitt Dresden Fakultt Informatik June 15, 2006 1 The Problem 2 The Basics 3 The Proposed Solution Learning by Machines Learning Rote Learning Reinforcement feedback

732 views • 34 slides
Humans vs Machines: When to Hire and When to Automate Lessons I Have Learned Managed Security

Humans vs Machines: When to Hire and When to Automate Lessons I Have Learned Managed Security

Humans vs Machines: When to Hire and When to Automate Lessons I Have Learned Managed Security Services Fog Creek Data Security Trello Agenda Evaluation Process Purchasing Process Renewal Process What I Would Do Sooner Demos /

356 views • 17 slides
Compressive Extreme Learning Machines Improved Models Through Exploiting Time-Accuracy Trade-offs

Compressive Extreme Learning Machines Improved Models Through Exploiting Time-Accuracy Trade-offs

Compressive Extreme Learning Machines Improved Models Through Exploiting Time-Accuracy Trade-offs Mark van Heeswijk, Amaury Lendasse, Yoan Miche September 5, 2014 Outline Motivation Extreme Learning Machines Compressive Extreme Learning

975 views • 48 slides
Virtual Machines Uses for Virtual Machines There are several uses for virtual machines:

Virtual Machines Uses for Virtual Machines There are several uses for virtual machines:

Virtual Machines Uses for Virtual Machines There are several uses for virtual machines: Virtual machine technology, often just called virtualization , makes one computer behave as several Running several operating systems simultaneously on

254 views • 10 slides
Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines?

Deep Reinforcement Learning Lecture 1 Sergey Levine How do we build intelligent machines? Intelligent machines must be able to adapt Deep learning helps us handle unstructured environments Reinforcement learning provides a formalism for

1.89k views • 172 slides
Learning Mealy Machines with Timers Bengt Jonsson Frits Vaandrager Uppsala University and

Learning Mealy Machines with Timers Bengt Jonsson Frits Vaandrager Uppsala University and

Introduction Mealy machines with timers Untimed semantics Learning algorithm Conclusions and future work Learning Mealy Machines with Timers Bengt Jonsson Frits Vaandrager Uppsala University and Radboud University Nijmegen IPA Fall Days,

769 views • 48 slides
? 17.10.2018 3 17.10.2018 4 Support Vector Machines (SVM): Background Support Vector Machines

? 17.10.2018 3 17.10.2018 4 Support Vector Machines (SVM): Background Support Vector Machines

INF3490 - Biologically inspired computing Support Vector Machines Support Vector Machines, Ensemble Learning, and Dimensionality (SVM) Reduction Weria Khaksar October 17, 2018 17.10.2018 2 Support Vector Machines (SVM): Background Support

579 views • 9 slides
Music recommenda tion System - Spotify Collaborative Filtering and Feedback System 1 Mithun

Music recommenda tion System - Spotify Collaborative Filtering and Feedback System 1 Mithun

Music recommenda tion System - Spotify Collaborative Filtering and Feedback System 1 Mithun Madathil 2 Table of contents Introduction Methods of recommendation Collaborative Filtering in Spotify Feedback System

2.29k views • 22 slides
Music recommendation at Spotify Ben Carterette What we do Spotifys mission is to unlock the

Music recommendation at Spotify Ben Carterette What we do Spotifys mission is to unlock the

Music recommendation at Spotify Ben Carterette What we do Spotifys mission is to unlock the potential of human creativity by giving a million creative artists the opportunity to live off their art and billions of fans the

481 views • 31 slides
Learning Machines Seminars 2020-11-05 Uncertainty in deep learning Olof Mogren, PhD RISE

Learning Machines Seminars 2020-11-05 Uncertainty in deep learning Olof Mogren, PhD RISE

Learning Machines Seminars 2020-11-05 Uncertainty in deep learning Olof Mogren, PhD RISE Research Institutes of Sweden Our world is full of uncertainties: measurement errors, modeling errors, or uncertainty due to test-data being

325 views • 31 slides

More recommend