Netflix’s Viewing Data Microservices How We Know Where You Are in House of Cards @zimmermatt
Netflix Scale @zimmermatt
Netflix Scale 62,000,000 @zimmermatt
Netflix Scale 62,000,000 50+ @zimmermatt
Netflix Scale 62,000,000 50+ 3,000,000,000+ @zimmermatt
Netflix Scale 62,000,000 50+ 3,000,000,000+ 1000+ @zimmermatt
Netflix Scale 62,000,000 50+ 3,000,000,000+ 1000+ 37% @zimmermatt
Viewing Data @zimmermatt
Viewing Data: History Who @zimmermatt
Viewing Data: History Who, What @zimmermatt
Viewing Data: History Who, What, When 6/10/2015 11:55 @zimmermatt
Viewing Data: History Who, What, When, Where 6/10/2015 11:55 @zimmermatt
Viewing Data: History Who, What, When, Where, How Long 6/10/2015 11:55 @zimmermatt
Viewing Data: Telemetry @zimmermatt
What have I watched? @zimmermatt
What have I watched? @zimmermatt
Where was I at? @zimmermatt
What else am I watching? @zimmermatt
What Happened? Video Bitrate Usage @zimmermatt
Viewing Data Requests @zimmermatt
Viewing Data Requests @zimmermatt
Viewing Data Requests @zimmermatt
Viewing Data Requests @zimmermatt
Viewing Data Requests @zimmermatt
Gen 3 (Existing) System Stateful Tier Events 0 Viewing Service 1 Active Sessions Summarized … Sessions Latest Positions n-2 n-1 History Cache Session Summary Logs Stateless Tier Viewing History (History, logs, and fallback) @zimmermatt
Gen 3 System’s Architecture & Scale Logs @zimmermatt
Gen 3 System’s Architecture & Scale Logs Viewing History @zimmermatt
Gen 3 System’s Architecture & Scale History Cache Logs Viewing History @zimmermatt
Gen 3 System’s Architecture & Scale Summarized Sessions History Cache Logs Viewing History @zimmermatt
Gen 3 System’s Architecture & Scale Stateful Tier Viewing Service Summarized … Sessions History Cache Logs Stateless Tier Viewing History (History, logs, and fallback) @zimmermatt
Gen 3 System’s Architecture & Scale Stateful Tier Events 0 Viewing Service 1 Active Sessions Summarized … Sessions Latest Positions n-2 n-1 History Cache Session Summary Logs Stateless Tier Viewing History (History, logs, and fallback) @zimmermatt
Growth 2015 @zimmermatt
Virtuous Cycle Improved Viewing Personalization Better Experience @zimmermatt
Stateful Instance Count Over Time @zimmermatt
Guiding Lights
Externalize State… Stateful Tier Events 0 Viewing Service 1 Active Sessions Summarized … Sessions Latest Positions n-2 n-1 History Cache Session Summary Logs Stateless Tier Viewing History (History, logs, and fallback) @zimmermatt
Externalize State… DENIED! Stateful Tier Events 0 Viewing Service 1 Active Sessions Summarized … Sessions Latest Positions n-2 n-1 History Cache Session Summary Logs Stateless Tier Viewing History (History, logs, and fallback) @zimmermatt
Service Decomposition Video Bitrate Usage @zimmermatt
Service Decomposition Start Stop @zimmermatt
Service Decomposition Start Stop Active Last Viewing Data Sessions Position History Feed @zimmermatt
Service Decomposition Start Stop Event Stream Session Stream State Summary Active Last Viewing Data Sessions Position History Feed @zimmermatt
Service Decomposition Events Collect Process Queries Provide @zimmermatt
Gen 3 Review Stateful Tier Events 0 Viewing Service 1 Active Sessions Summarized … Sessions Latest Positions n-2 n-1 History Cache Session Summary Logs Stateless Tier Viewing History (History, logs, and fallback) @zimmermatt
Gen 4 Generic Architecture Collector Processor Provider @zimmermatt
Gen 4 Generic Architecture Collector Processor Provider Queue @zimmermatt
Gen 4 Generic Architecture Collector Processor Provider Queue Raw Data Materialized Views @zimmermatt
Gen 4 Generic Architecture Collector Processor Provider Cache Queue Raw Data Materialized Views @zimmermatt
Gen 4 Architecture Positions Session Events Viewing History Collector Playback History Provider Log Session Accounting Collector Provider Event Data Session Summary Raw Data Processor History Raw Data Logs Active Raw Data Sessions Materialized View Summarized Sessions Materialized View @zimmermatt
Shadow Testing Session Summary Log Session Events Collector Collector Processor Stateful Tier Events 0 Viewing Service Logs Summarized Event Data Active Sessions 1 Sessions … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Positions Viewing History Stateless Tier (History, logs, Active Playback History and fallback) Provider Sessions History Cache History Viewing History Summarized Logs Sessions @zimmermatt
Shadow Testing: Start Play Event Session Events Collector Stateful Tier Events 0 Viewing Service Event Data Active Sessions 1 … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Stateless Tier (History, logs, Active and fallback) Sessions @zimmermatt
Shadow Testing: Start Play Event Session Events Collector Stateful Tier Events 0 Viewing Service Event Data Active Sessions 1 … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Stateless Tier (History, logs, Active and fallback) Sessions @zimmermatt
Shadow Testing: Start Play Event Session Events Collector Stateful Tier Events 0 Viewing Service Event Data Active Sessions 1 … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Stateless Tier (History, logs, Active and fallback) Sessions @zimmermatt
Shadow Testing: Start Play Event Session Events Collector Stateful Tier Events 0 Viewing Service Event Data Active Sessions 1 … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Stateless Tier (History, logs, Active and fallback) Sessions @zimmermatt
Shadow Testing: Start Play Event Session Events Collector Stateful Tier Events 0 Viewing Service Event Data Active Sessions 1 … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Stateless Tier (History, logs, Active and fallback) Sessions @zimmermatt
Shadow Testing: Read Latest Position Stateful Tier Events 0 Viewing Service Active Sessions 1 … Edge Service Latest Positions n-2 n-1 Session Summary Positions Viewing History Stateless Tier (History, logs, Playback History and fallback) Provider History @zimmermatt
Shadow Testing: Read Latest Position Stateful Tier Events 0 Viewing Service Active Sessions 1 … Edge Service Latest Positions n-2 n-1 Session Summary Positions Viewing History Stateless Tier (History, logs, Playback History and fallback) Provider History @zimmermatt
Shadow Testing: Read Latest Position Stateful Tier Events 0 Viewing Service Active Sessions 1 … Edge Service Latest Positions n-2 n-1 Session Summary Positions Viewing History Stateless Tier (History, logs, Playback History and fallback) Provider History @zimmermatt
Shadow Testing: Stop Event Session Summary Session Events Collector Processor Stateful Tier Events 0 Viewing Service Summarized Event Data Active Sessions 1 Sessions … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Positions Viewing History Stateless Tier (History, logs, Active Playback History and fallback) Provider Sessions History Summarized Sessions @zimmermatt
Shadow Testing: Stop Event Session Summary Session Events Collector Processor Stateful Tier Events 0 Viewing Service Summarized Event Data Active Sessions 1 Sessions … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Positions Viewing History Stateless Tier (History, logs, Active Playback History and fallback) Provider Sessions History Summarized Sessions @zimmermatt
Shadow Testing: Stop Event Session Summary Session Events Collector Processor Stateful Tier Events 0 Viewing Service Summarized Event Data Active Sessions 1 Sessions … Edge Service Latest Positions n-2 Session Accounting n-1 Provider Session Summary Positions Viewing History Stateless Tier (History, logs, Active Playback History and fallback) Provider Sessions History Summarized Sessions @zimmermatt
Recommend
More recommend