the many flavors Keith Stobie Doyenz (while at Microsoft) Copies - PowerPoint PPT Presentation

Testing In Production the many flavors Keith Stobie Doyenz (while at Microsoft) Copies may not be made or distributed for commercial use

Pre-Requisites (risk reduction) • Redundancy – state quickly recovered if catastrophic failure, • Data corruption or state-related failures extremely unlikely ( functional testing must first be passing), • Errors must be detected and the engineering team (rather than operations) must be monitoring system health of the code in test • Can quickly roll back all changes Rollback tested before going into production 2011-10-10 PNSQC Testing in Production -- Keith Stobie 2 Copies may not be made or distributed for commercial use

Image from : http://blogs.msdn.com/b/seliot/archive/2011/06/07/testing-in-production-tip-it-really-happens-and-that-s-a-good-thing.aspx Isolate Test Data • Don’t let test data affect Production metrics. Can’t bill for test data! • Number of users requests must be distinct from test requests. • Is test data visible? Fake products? 2011-10-10 PNSQC Testing in Production -- Keith Stobie 3 Copies may not be made or distributed for commercial use

Exposure Control Growing visibility (exposure / risk ) • Opt-in: Preview, Beta, . . . • Google: Internal, Friends&Family, Beta, . . . • % of Machines/DCs : 1 %, 5%, 10%, 25% 50% • Markets: US, Europe, Asia, America 2011-10-10 PNSQC Testing in Production -- Keith Stobie 4 Copies may not be made or distributed for commercial use

Asserts == Testing • Leave asserts that don’t significantly impact performance. Assert(inputRef != null) (if not in frequent path) • Self-test, like hardware, some software has low-pri thread for idle times to walk and verify data structures, etc. 2011-10-10 PNSQC Testing in Production -- Keith Stobie 5 Copies may not be made or distributed for commercial use

Monitoring == Testing % of requests Threshold • Example: In Testbed, 50% 100ms is the response time 90% 200ms < threshold(s)? 99% 300ms • Monitor same thresholds in production: – External “synthetic” transactions – Real time data collection • Transactional availability • Feature availability • “pure” monitor is a test Oracle 2011-10-10 PNSQC Testing in Production -- Keith Stobie 6 Copies may not be made or distributed for commercial use

Delta Assertion (Meszaros) Image from : http://xunitpatterns.com/Delta%20Assertion.html 2011-10-10 PNSQC Testing in Production -- Keith Stobie 7 Copies may not be made or distributed for commercial use

Model of a Service Backend User Frontend Service service Requests service Data Data Feed 2011-10-10 PNSQC Testing in Production -- Keith Stobie 8 Copies may not be made or distributed for commercial use

(Offline) Test in Production Environment Backend User Frontend Service service Requests service Data Data Feed Production Environment (Identical Hardware, operations, etc.) Test 2011-10-10 PNSQC Testing in Production -- Keith Stobie 9 Copies may not be made or distributed for commercial use

Fork(copy) Traffic (drop test return) User Frontend Backend Service Requests service service Productio Data Production n Data Feed Test Test Backend Service service Data Data Feed 2011-10-10 PNSQC Testing in Production -- Keith Stobie 10 Copies may not be made or distributed for commercial use

Experiment with Graphics/Color • Which one converts (to search) better? A B Source: Marketing Experiments 2011-10-10 PNSQC Testing in Production -- Keith Stobie 11 http://www.marketingexperiments.com Copies may not be made or distributed for commercial use

A/B Test 50% 50% Visitors Randomly Distributed Version A Version B (Control) (Treatment) Page Title Title of Page Signup Signup Here Here Click Thru Rate R R T T User interactions instrumented, C C analyzed and compared 1.2% of users with a Page 2.8% of users with a Page View clicked on Signup View clicked on Signup Is the observed difference statistically significant? Figure courtesy of Seth Elliot from his SASQAG (Seattle Area Software Quality Assurance Group) April YES 2011 Talk Testing in Production - Your Key to Engaging Customers Version B is better than Version A 2011-10-10 PNSQC Testing in Production -- Keith Stobie 13 Copies may not be made or distributed for commercial use

Experiments (alternate return) User Frontend Backend Service Requests service service Data Production Data Feed Experiment Service Separate experimental service providing the 2 nd alternate output 2011-10-10 PNSQC Testing in Production -- Keith Stobie 14 Copies may not be made or distributed for commercial use

Experiments (alternate code) Service User Frontend Backend Requests service service Service Data Production Data Feed Experiment Service architected internally to have code paths for different alternate output 2011-10-10 PNSQC Testing in Production -- Keith Stobie 15 Copies may not be made or distributed for commercial use

Experiments (alternate data) Service User Frontend Backend Requests service service Data Production Data Data Feed Experiment Sometimes it is just the data, e.g. different ranking values that are being tried. 2011-10-10 PNSQC Testing in Production -- Keith Stobie 16 Copies may not be made or distributed for commercial use

Shadowing (compare returns) User Frontend Backend Service Requests service service Data Data Feed Production Environment Future Equally Backend Environment good? Service service Production Next Environ Data Data Feed Test 2011-10-10 PNSQC Testing in Production -- Keith Stobie 17 Copies may not be made or distributed for commercial use

Fake Traffic (capacity test) User Frontend Backend Service Requests service service replayed Requests Data Data Feed Production Test Capacity test a single service, or entire offering (via the frontend) 2011-10-10 PNSQC Testing in Production -- Keith Stobie 18 Copies may not be made or distributed for commercial use

From: Greg Veith – Workshop on Performance & Reliability 2010 (WOPR14) Modern Cap Generation Toolset DC1 DC3 Monitoring system Monitoring system Capacity clients Capacity clients VI VI P P Bing Entry point Bing entry point Controller DC…n DC2 Monitoring system Monitoring system Capacity clients Capacity clients VI VI P P Bing entry point Bing Entry point 1. Controller drives each of the clients across any/all DCs 2. Capacity is generated within each datacenter to avoid cross-DC network utilization 2011-10-10 PNSQC Testing in Production -- Keith Stobie 19 3. Controller pulls data from monitoring system auto-throttling based on SLA Copies may not be made or distributed for commercial use 4. 2 -3 engineers required for a typical full scale test

Tracers (data flow) User Frontend Backend Service Requests service service Test Requests Data Data Feed Production Data Feed Test Observe test data as it flows across components and out of the system. 2011-10-10 PNSQC Testing in Production -- Keith Stobie 20 Copies may not be made or distributed for commercial use

Shopping Test Product Product in Feed? Catalog Trace Test Product Product Product Authority Data Product Matching Test Offer Test Offer in Feed? Offers & Reviews Merchant Self matched to Offers Service Products? Portal Aggregator Test Review Offers & Reviews Crawl counts correct? Pipeline Test Review 1. Product search page displays in Feed? product when users queries for it? Production 2. Review search page displays User reviews when users queries for it? Test Serving 3. Product details page for a given product contains matched offer and 2011-10-10 PNSQC Testing in Production -- Keith Stobie 21 review information? Copies may not be made or distributed for commercial use

Fault Injection Loss of Service Loss of Service (or machine or (or machine or … ) … ) Service User Frontend Backend Service Requests service service Data corruption Network latency Data Data Feed Productio Production n Test Test Image from : http://behance.vo.llnwd.net/profiles/78972/projects/114549/789721221731305.jpg 2011-10-10 PNSQC Testing in Production -- Keith Stobie Copies may not be made or distributed for commercial use

Summary Risk of Causing Exposure Data Approach issues Control Control Asserts Medium Optional -- Monitoring Low -- -- Offline Low -- -- Controlled Medium Required Useful Experiment Live Capacity High Useful Required Tracers Medium Useful Useful Fault Injection High Optional Useful 2011-10-10 PNSQC Testing in Production -- Keith Stobie 23 Copies may not be made or distributed for commercial use

Testing in Production 2011-10-10 PNSQC Testing in Production -- Keith Stobie 24 Copies may not be made or distributed for commercial use