How Netflix directs 1/3rd of Haley Tucker QCon San Francisco Mohit Vora Nov 16, 2015
Playback Overview
DATA PLANE NETFLIX STREAM DEVICE (CDN) CONTROL PLANE
Project 366 #59; 280212 Days Gone By..., CC BY-SA, Pete 2012, Flickr
VIDEO AUDIO TEXT STREAMS
How do we build a streaming “tape”?
Determine the preferred experience TITLE COUNTRY DEVICE CUSTOMER NETWORK CONNECTIONS Broadband - wired or wifi Cellular - Edge, 3G, LTE, ...
That’s exactly what I want ...now where can I get it?
Point the device to appropriate locations Steering
PLAYBACK MANIFEST GENERATE PLAYBACK PLAYBACK MANIFEST MANIFEST
Uh-oh, the content is encrypted! Keymaster, CC BY-SA, Sean McGrath 2007, Flickr
LICENSE LICENSE
And...Action!
SESSION EVENTS SESSION (START, STOP, PAUSE, RESUME, KEEPALIVE)
PLAYBACK LIFECYCLE GENERATE PLAYBACK PLAYBACK MANIFEST MANIFEST LICENSE SESSION (START, STOP, PAUSE, RESUME, KEEPALIVE)
Data Plane (CDN)
What is a Content Delivery Network?
Open Connect A NETFLIX ORIGINAL
D E M A S E E R T T Y S B CONTENT RANK PREDICTABLE VIEWING PATTERNS
FILLING WHEN YOU SLEEP Dreaming…, CC BY-SA, Eleni Boulsaiki 2009, Flickr
ONE WAY, CC BY-SA, Kenny Louie 2010, Flickr Open Connect A NETFLIX ORIGINAL READ XOR WRITE FILLING WHEN YOU SLEEP
Content Delivery Mechanisms
DATA PLANE NETFLIX STREAM DEVICE (CDN) CONTROL PLANE
NETFLIX ISP STREAM ROUTER DEVICE ISP DATA CENTER
NETFLIX ISP STREAM ROUTER DEVICE ISP DATA CENTER ISP CO-LOCATION
NETFLIX ISP STREAM ROUTER DEVICE ISP DATA CENTER
IXP DATA ISP CENTER ROUTER NETFLIX ISP STREAM ROUTER DEVICE NFLX ROUTER ISP DATA CENTER NETFLIX
IXP DATA ISP CENTER ROUTER NETFLIX ISP STREAM ROUTER DEVICE NFLX ROUTER ISP DATA CENTER NETFLIX
IXP DATA ISP CENTER ROUTER NETFLIX ISP STREAM ROUTER DEVICE NFLX ROUTER ISP DATA CENTER NETFLIX IXP INTERCONNECTION
Control Plane
OPEN CONNECT NETFLIX STREAM DEVICE CDN Network Proximity DEVICE CONTROL CONTROL DON’T KEEP SECRETS Content Positioning PLANE PLANE Load Distribution
Network Proximity Social Network in a Course, CC BY-SA, Hans Põldoja 2010, Flickr
By Specification?
By Specification? Doesn’t scale
TAKEAWAY Use BGP Border Gateway Protocol BGP ROUTE 175.231.128.0/24 (+ proximity attributes)
IXP DATA ISP1 DATA CENTER CENTER ISP1 ISP2 DATA CENTER NFLX ISP2 BGP ROUTES ISP1 BGP ROUTES BGP ROUTE CONTROL 175.231.128.0/24 PLANE (+ proximity attributes)
Content Positioning
SERVE CACHE MISS ISP DATA CENTER LOCALIZE TRAFFIC
HOW DO WE DETERMINE WHAT CONTENT WILL BE POPULAR TOMORROW?
CHANGING CATALOG
EVOLVING MEMBER TASTES
ISP OFF PEAK DATA CENTER FILL MINIMIZE FILL CHURN
D E M A S E E R T T Y B S CONTENT RANK USE HISTORICAL DATA bytesStreamed/bytesStored
IS ONE DAY OF HISTORY ENOUGH?
TAKEAWAY Weigh Recent Data Higher � � � � � … WEIGHT � = 0.9 0 10 20 30 40 DAYS AGO EXPONENTIALLY WEIGHTED MOVING AVERAGE
HOW SHOULD CONTENT BE ALLOCATED?
HOW SHOULD CONTENT BE ALLOCATED? MILLIONS THOUSANDS OF FILES OF SERVERS
TAKEAWAY Consistent Hashing SVR3 FILE1 SVR4 FILE1 SVR1 SVR2 FILE3 ALLOCATE MULTIPLE REPLICAS RESILIENT TO CLUSTER CHANGES REPEATABLE
IXP DATA ISP1 DATA CENTER CENTER ISP2 DATA WHAT TO FILL? CENTER FILL OVER HTTP WHERE TO FILL FROM? CONTROL S3 PLANE
Load Distribution
LOTS OF THROUGHPUT STREAMED BYTES LOTS OF STORAGE CONTENT RANK CONTENT WITH CONFLICTING CONSTRAINTS
TAKEAWAY Tier Infrastructure MEMORY SSD S P I N N I N G D I S K SSD BASED SPINNING DISK STREAMED BASED BYTES CONTENT RANK WITHIN CLUSTERS ON EACH SERVER
HOW DO WE BALANCE LOAD? BALANCE BALANCE ACROSS EQUIDISTANT ACROSS SERVERS CLUSTERS WITHIN CLUSTERS
OPEN CONNECT NETFLIX STREAM DEVICE LOAD BALANCER CDN DEVICE CONTROL CONTROL PLANE PLANE
HOW DO WE BALANCE LOAD? USING CONTENT DISTRIBUTION
AND WHEN WE HAVE EQUALLY ATTRACTIVE LOCATIONS TO SERVE FROM – FLIP A COIN
MAX S M C E I R T T S SANE INSANE E Y M S INCIDENT LOAD HOW DO WE LOAD SERVERS OPTIMALLY?
… AMIDST EVER CHANGING INTERNET WEATHER
TRAFFIC t … AND DAILY TRAFFIC EBBS AND FLOWS
CONTROL + TRAFFIC EFFECT ON SERVE - SYSTEM METRICS STREAMS FEEDBACK WE INTRODUCE A FEEDBACK LOOP
TAKEAWAY PID CONTROLLER
TAKEAWAY PID CONTROLLER Process Current RPM System Metrics Variable System Metrics Set Point Desired RPM Max Control Controlled Input Voltage Traffic Variable DC MOTOR
TAKEAWAY PID CONTROLLER Process Current RPM System Metrics Variable System Metrics Set Point Desired RPM Max Control Controlled Input Voltage Traffic Variable LOADING SERVERS
IXP DATA ISP1 DATA CENTER CENTER ISP2 DATA CONTROL CENTER TO 80% NO CONTROL CONTROL 0.0 < CONTROL VAR < 1.0 PLANE
NEXT HOP TRAFFIC t TRAFFIC SHIFTS TO NEXT HOP LOCATION
Steering
OPEN CONNECT NETFLIX STREAM DEVICE KAFKA Yes, here’s the URLs CDN STEERING CONTROL PLAYBACK Got URLs for PLANE SERVICES f1, f2, …, fn? CASS PROXIMITY HEALTH CONTENT
Architecture Evolution 5 CHALLENGES
How did we evolve from here... MANIFEST STEERING SESSION API LICENSE DRM
...to here. MANIFEST CLIENT SCRIPTS STEERING RULES INSIGHTS SESSION SERVICE LAYER CACHE API LICENSE DRM 5 SOLUTIONS
High dimensionality CHALLENGE TITLE COUNTRY DEVICE CUSTOMER NETWORK CONNECTIONS Broadband - wired or wifi Cellular - Edge, 3G, LTE, ...
How can we quickly alter the playback experience in a targeted manner?
USE CASE Stream Filtering RULES ENGINE ALL BEST STREAMS STREAMS FOR FOR CONTENT SESSION
EXAMPLE RULES
UPDATING RULES PUBLISH CONFIGURATION MANAGEMENT UI TOPIC SUBSCRIBE RULES ENGINE
TAKEAWAY Dynamic Business Rules MANIFEST STEERING RULES API SESSION LICENSE DRM
CHALLENGE Pinpoint what is broken Haystacks, CC BY-SA, John Pavelka 2008, Flickr
3:00 AM : Pager goes off
METRICS AND ALERTING
OK...error code 105 is elevated. But why?
Indexed Logging
TAKEAWAY Detailed Domain Insights MANIFEST STEERING RULES INSIGHTS API SESSION LICENSE DRM
Large amount of state CHALLENGE
How can we enable faster UIs and low-end devices?
We introduced a server-side caching tier MANIFESTS CUSTOMER A CUSTOMER B CUSTOMER A
Watch out for resiliency issues!! Ping Pong project, CC BY-SA, Michael Knowles 2008, Flickr
TAKEAWAY Reduce client state MANIFEST STEERING RULES INSIGHTS API SESSION CACHE LICENSE DRM
Managing device protocols CHALLENGE Square peg, round hole, CC BY-SA, Simon Law 2006, Flickr
Recommend
More recommend
