Crisis to Calm: A Story of Data Validation @ Netflix Lavanya - PowerPoint PPT Presentation

Mar 17, 2024 •232 likes •778 views

Crisis to Calm: A Story of Data Validation @ Netflix Lavanya Kanchanapalli Rollback data Increase capacity Netflix starts to work Root Cause Duplicate objects Apps failed Cascading failures Netflix goes down! Data

Crisis to Calm: A Story of Data Validation @ Netflix Lavanya Kanchanapalli
● Rollback data ● Increase capacity ● Netflix starts to work
Root Cause ● Duplicate objects ● Apps failed ● Cascading failures ● Netflix goes down!
Data Change Netflix Microservices Cloud App1 Appn App2 Change in Behavior
Metadata Architecture Traffic Netflix Netflix Source Services Netflix Services Netflix System Services Netflix Services Video Service Metadata Service Source System Amazon S3
● Single publisher ● Multiple consumers ● Hollow (hollow.how) ● Versioned data ● Fast propagation
Bad data happens!
Leaked Content
Disables Features
Deletes data
Data change = Code Push
1 2 2 3 3 Detection Staggering Staggering Rollback Rollback
1 2 3 3 Detection Staggering Rollback Rollback
1 2 3 Detection Staggering Rollback
Many Unknowns
Would it be too slow?
Would there be too many failures?
Would it cost too much?
1 2 3 Detection Staggering Rollback
1 2 2 3 3 Detection Staggering Staggering Rollback Rollback
Chapter 1: Circuit Breakers
Metadata Architecture Traffic Netflix Netflix Source Services Netflix Services Netflix System Services Netflix Services Video Service Metadata Service Source System Amazon S3
Circuit Breakers ● Integrity checks ● Duplicate detection ● Object counts ● Semantic checks
Know your data change
Knobs are key to sanity ● On/ off ● Threshold ● Exclusions
Business value is the key
Efficiency ● Change Isolation ● Sampling
Chapter 2: Canaries
Traditional Canaries Traffic Shadow Shadow Netflix Traffic Traffic Netflix Source Services Netflix Services Netflix System Services Netflix Services Video Service Metadata Service Baseline Canary Source (Old Code) (New Code) System Amazon S3
Data Canaries Traffic Netflix Netflix Data Netflix Service Canary Netflix Services Netflix Source Service Services Netflix System Services Netflix Services Video Service Metadata Service Source System Amazon S3
Netflix Data Canary Service ● Pick key use case(s) Netflix Data Netflix Service ● Pick data to test Canary Service ● Test with latest data
1 2 3 Detection Staggering Rollback Rollback
Amazon WS Global Infrastructure STAGGERED ROLLOUT
1 2 3 Detection Staggering Rollback
Keep calm & rollback ● Pin back ● Root cause ● Unpin
Rollback ● Visibility ● Traversing
Data Diff UI
Circuit Breaker UI
Pinning UI
Data validation is key to high availability ● Data change = Code push ● Circuit breakers & canaries ● Staggering and rollback
Thank You

Recommend

CALM Sueann Tupy, Educator AGENDA Grounding Activity The Story of CALM Our Vision Program

SHAC Meeting March 20, 2019 CALM Sueann Tupy, Educator AGENDA Grounding Activity The Story of CALM Our Vision Program Goals Staff & Community Teacher Feedback Summary Grounding Activity What is CALM? C hildren and C ommunities A

396 views • 18 slides

Containers at Netflix - An Evolving Story Sangeeta Narayanan Engineering Manager @Netflix

Containers at Netflix - An Evolving Story Sangeeta Narayanan Engineering Manager @Netflix @sangeetan Going Global Recent Launches: Italy, Spain, Portugal Early 2016: Korea, Hong Kong, Taiwan, Singapore Growing BIGGER Winning Moments of

865 views • 61 slides

8 WAYS TO STAY CALM IN A CRISIS! www.bulletproofactor.com www.actonthis.tv #CORONAVIRUS THERE

8 WAYS TO STAY CALM IN A CRISIS! www.bulletproofactor.com www.actonthis.tv #CORONAVIRUS THERE IS A LOT OF FEAR AND UNCERTAINTY IN SOCIETY RIGHT NOW - AND RIGHTLY SO. THE SPREAD OF COVID-19 HAS TAKEN US ALL DOWN A PATH WE NEVER COULD

316 views • 19 slides

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides

Netflix: Netflix: Petabyte Scale Petabyte Scale Analytics Infrastructure in Analytics

Netflix: Netflix: Petabyte Scale Petabyte Scale Analytics Infrastructure in Analytics Infrastructure in the Cloud the Cloud Daniel C. Weeks Tom Gianos Overview Data at Netflix Netflix Scale Platform Architecture Data

887 views • 60 slides

Netflix: Integrating Spark At Petabyte Scale Ashwin Shankar Cheolsoo Park Outline 1. Netflix

Netflix: Integrating Spark At Petabyte Scale Ashwin Shankar Cheolsoo Park Outline 1. Netflix big data platform 2. Spark @ Netflix 3. Multi-tenancy problems 4. Predicate pushdown 5. S3 file listing 6. S3 insert overwrite 7. Zeppelin, Ipython

526 views • 52 slides

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides

How We Know Where You Are in House of Cards @zimmermatt Netflix Scale @zimmermatt Netflix

Netflixs Viewing Data Microservices How We Know Where You Are in House of Cards @zimmermatt Netflix Scale @zimmermatt Netflix Scale 62,000,000 @zimmermatt Netflix Scale 62,000,000 50+ @zimmermatt Netflix Scale 62,000,000 50+

1.07k views • 90 slides

1 QC STORY -32 QC STORY -32 QC STORY -32 QC Story-1 QC Story-1 QC Story-1 Awards and

QC STORY -32 QC STORY -32 QC STORY -32 Shop Floor view QC Story-1 QC Story-1 FACTORY OVERVIEW QC Story-1 On Behalf of Susira Industries Ltd., Chennai. We Welcome the Delegates for the QCI DL SHAH NATIONAL AWARDS ON ECONOMICS OF

976 views • 13 slides

Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien

Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien Zhang, ZTE How would OPNFV attract more open source software developers ? 2 I wrote the script based on the instructions in ETSI NFV documents, but

293 views • 15 slides

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides

Calm Computing The Coming Age of Mark Weiser and John Seely Brown Calm Computing Whyfor, Calm

Calm Computing The Coming Age of Mark Weiser and John Seely Brown Calm Computing Whyfor, Calm Computing? Whyfor, Calm Computing? Computer Evolution Whyfor, Calm Computing? Computer Evolution Mainframes Whyfor, Calm Computing? Computer

1.08k views • 95 slides

Calm Technology Calm Technology Design for the Next Generation of Devices Amber Case |

Calm Technology Calm Technology Design for the Next Generation of Devices Amber Case | @caseorganic Research Fellow, Harvard Berkman Klein Center, @medialab QCon SF | 9 Nov 2016 calmtech.com caseorganic.com 50 billion devices 50 billion will

688 views • 47 slides

Update on aircraft validation efforts; T/q retrieval validation using ARM data Dave Tobin, Leslie

Update on aircraft validation efforts; T/q retrieval validation using ARM data Dave Tobin, Leslie Moy, Bob Knuteson, Hank Revercomb, Fred Best, Joe Taylor CIMSS / SSEC / UW-Madison AIRS Science Team Meeting Pasadena, CA 27-30 March 2007

425 views • 30 slides

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views • 58 slides

Data validation and exploration Data validation and exploration Abhijit Dasgupta Abhijit

Data validation and exploration Data validation and exploration Abhijit Dasgupta Abhijit Dasgupta Fall, 2019 Fall, 2019 1 BIOF339, Fall, 2019 Plan today Dynamic exploration of data Data validation Missing data evaluation 2 BIOF339,

570 views • 27 slides

Forms and Validation Topics: Sending data to the server forms HTML 5 validation

Forms and Validation Topics: Sending data to the server forms HTML 5 validation Server side <form> element Forms allow us to submit data to the server There are two ways of handling information submitted with a

644 views • 14 slides

Peering to Scale the Netflix Perspective Scaling for Growth How Does Netflix Manage Growth?

Peering to Scale the Netflix Perspective Scaling for Growth How Does Netflix Manage Growth? How Does Netflix Help You Manage Growth? What is My Peering Strategy The Future Scaling for Growth the Netflix Example Netflix Traffic

457 views • 20 slides

How Netflix directs 1/3rd of Haley Tucker QCon San Francisco Mohit Vora Nov 16, 2015 Playback

How Netflix directs 1/3rd of Haley Tucker QCon San Francisco Mohit Vora Nov 16, 2015 Playback Overview DATA PLANE NETFLIX STREAM DEVICE (CDN) CONTROL PLANE Project 366 #59; 280212 Days Gone By..., CC BY-SA, Pete 2012, Flickr VIDEO

1.26k views • 112 slides

From Silos to Circles: A Story of Crisis to Systemic Change Healing Families, Mending the

From Silos to Circles: A Story of Crisis to Systemic Change Healing Families, Mending the Circle GRAND RIVER SOCIAL SERVICES Nitumigaabow Ryan Champagne, MSW Jerome Brooks Big John, MTA RWJF Health Award 2015 Robert Woods Johnson

431 views • 13 slides

Welcome My name is Jennie Harrison and I am The Calm Mum s Coach. I work with Mums and Little

Sleepy Biz Mums Welcome My name is Jennie Harrison and I am The Calm Mum s Coach. I work with Mums and Little Ones helping them to have less stress, more sleep and to help Mums be the calm mum they want to be Jennie Harrison The Calm Mum

332 views • 9 slides

Privacy Preserving Data Mining Li Xiong Department of Mathematics and Computer Science

CS378 Introduction to Data Mining Privacy Preserving Data Mining Li Xiong Department of Mathematics and Computer Science Department of Biomedical Informatics Emory University Netflix Sequel 2006, Netflix announced the challenge 2007,

951 views • 58 slides

Improving Netflix Performance Bill Scott Director, UI Engineering Netflix June 23, 2008 1

Improving Netflix Performance Bill Scott Director, UI Engineering Netflix June 23, 2008 1 The Plan Accurately measure the full user experience performance Prior Page Request Response Page Web App Incrementally improve site

1.04k views • 24 slides

!"#$!#!%& Critical thinking Validation = critical assessment How good is my

!"#$!#!%& EMBO Course on SAS, EMBL-HH, 2 November 2014 Applied common sense The why, what and how of validation Gerard J. Kleywegt Protein Data Bank in Europe (pdbe.org) EMBL-EBI, Cambridge, UK Validation according to the dictionary

407 views • 19 slides