¡ ¡ T18 ¡ Performance ¡Testing ¡ Thursday, ¡October ¡3rd, ¡2019 ¡1:30 ¡PM ¡ ¡ ¡ ¡ ¡ Continuous ¡Performance ¡Testing: ¡A ¡ Tale ¡of ¡Two ¡Worlds ¡ ¡ Presented ¡by: ¡ ¡ ¡ ¡ Kaushal ¡Dalvi ¡ ¡ Ultimate ¡Software ¡ ¡ ¡ Brought ¡to ¡you ¡by: ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ 888 -‑-‑-‑ 268 -‑-‑-‑ 8770 ¡ ·√·√ ¡904 -‑-‑-‑ 278 -‑-‑-‑ 0524 ¡-‑ ¡info@techwell.com ¡-‑ ¡http://www.starwest.techwell.com/ ¡ ¡ ¡ ¡ ¡ ¡ ¡
¡ Kaushal ¡Dalvi ¡ ¡ Since ¡he ¡got ¡his ¡first ¡computer ¡at ¡age ¡thirteen, ¡Kaushal ¡Dalvi ¡has ¡been ¡interested ¡in ¡ systems ¡and ¡software ¡performance. ¡He ¡spent ¡days ¡researching ¡performance ¡ characteristics ¡of ¡different ¡motherboards, ¡CPUs, ¡GPUs, ¡RAM, ¡and ¡disks ¡to ¡configure ¡ and ¡overclock ¡them ¡in ¡order ¡to ¡squeeze ¡the ¡maximum ¡frames ¡per ¡second ¡out ¡of ¡the ¡ games ¡he ¡played. ¡Kaushal ¡built ¡and ¡maintained ¡websites ¡for ¡local ¡businesses, ¡where ¡ he ¡started ¡learning ¡about ¡performance ¡and ¡reliability. ¡He ¡developed ¡a ¡taste ¡for ¡it, ¡ and ¡now ¡ten ¡years ¡later ¡he ¡continues ¡working ¡in ¡the ¡same ¡field ¡as ¡the ¡director ¡of ¡ quality ¡and ¡performance ¡engineering ¡at ¡Ultimate ¡Software, ¡leading ¡teams ¡that ¡deal ¡ with ¡workload ¡modeling, ¡test ¡scripting ¡and ¡execution, ¡bottleneck ¡analysis, ¡ debugging, ¡and ¡much ¡more. ¡ ¡
9/29/19 CONTINUOUS PERFORMANCE TESTING: A TALE OF TWO WORLDS -KAUSHAL DALVI WHAT IS PSR A POSSIBLY BETTER WAY TO REFER TO THE UMBRELLA TERM RATHER THAN JUST PERFORMANCE PERFORMANCE The latency and utilization SCALABILITY characteristics of systems under different types of load. The ability of the system to maintain performance characteristics under growing load, given more resources. P S R RESILIENCY The ability of the system to maintain or restore its performance characteristics when subjected to failures of components. 1
9/29/19 TYPES OF PSR TESTS THE DIFFERENT TYPES OF TESTS UNDER THE PERFORMANCE UMBRELLA VOLUME LOAD Gradually increasing the volume of the data or in general the work objects that the system A steady load based on regular peaks needs to process under normal workloads to observed or expected in productions applied understand the impact of growth on the for a short duration (usually 1-3 hours) to performance characteristics of the system. understand the performance characteristics of the application in normal conditions. SCALABILITY ENDURANCE Any kind of load test run with the objective of understanding whether the system is capable Prolonging the workload from load of maintaining performance characteristics test to longer periods (usually 8-72 when subjected to increasing workloads while hours) to understand the behavior of being provided additional resources. the application over time. RESILIENCY STRESS Any kind of performance test run with the Increasing the regular workloads by a objective of understanding whether the factor of 2X, 3X and so on to understand system is capable of maintaining or restoring the characteristics of the system under performance characteristics when subjected increased demand. to failures of components. IMPORTANCE OF TARGETS With great power comes great responsibility The extra-ordinary ability to waste everyone’s time ! Running the wrong test can lead to false overconfidence. Or it can lead to chasing demons that do not exist in the real world. 2
9/29/19 TARGETS Knowing the terminology SLA SLA SLO SLO SLI SLI NFR NF SERVICE LEVEL NON FUNCTIONAL SERVICE LEVEL SERVICE LEVEL INDICATOR REQUIREMENT AGREEMENT OBJECTIVE A requirement that specifies criteria An official commitment that prevails An internally agreed upon An indicator that shows the current that can be used to judge the between a service provider and a objective that the team is level being achieved by the systems operation of a system, rather than client. committed to meet. under observation. specific behaviors. Targets (…) SLA An official/legal commitment that prevails between a service provider and a NFR SLI client. A requirement that An indicator that specifies criteria that can shows the current be used to judge the level being achieved operation of a system, by the systems under SLO rather than specific observation. behaviors. An internally agreed upon objective that the team is committed to meet. 3
9/29/19 WORKLOAD MODELLING Characterization of the work performed by the system Designing the right performance test to answer relevant questions about PSR characteristics Workload modeling identifies one or more workload profiles to be simulated against the tested application. The workload model then attempts to approximate real life usage scenario and includes different user types and characteristics. https://blog.smartbear.com/software-quality/workload-modeling-and-profiles-for-load-testing/ PURPOSE OF PERFORMANCE TESTS Not all performance tests are created equal A/B TESTS CAPACITY REGRESSION BENCHMARKING Also know as To determine the upper limits To check whether the Market facing tests to show of the system in terms of load, performance of the software how well the software can optimization/tuning testing, to find the best solution to a data, other factors has been degraded by the perform at realistic loads changes to the source code design/architecture problem 4
9/29/19 MICROSERVICES vs MONOLITHS HIGH LEVEL PSR RELEVANT DIFFERENCES MICROSERVICE MONOLITH SMALL SERVICES BIG BALL OF MUD INDEPENDANTLY DEPLOYABLE COMBINED DEPLOYMENT MULTI STACK SINGLE STACK NETWORK COMMUNICATION INTERNAL COMUNICATION SAGAS TRANSACTIONS ??? PSR TEST CHALLENGES - MICROSERVICES AS OPPOSED TO TRADITIONAL MONOLITHS TE TEST E T ENVIR IRONMENTS TS EVERY SERVICE HAS ITS OWN ENVIRONMENT LEADING TO PROBLEMS AROUND CONFIGURATION AND SCHEDULING AMONG OTHER THINGS CAPACI CA CITY TE TEST D T DATA TA EASIER TO KNOW THE CAPACITY OF EACH INDIVIDUAL COMPONENT, OVERALL SYSTEM, EVERY SERVICE ALSO HAS ITS OWN NOT SO MUCH DATABASE, MAKING TEST DATA MANAGEMENT A NIGHTMARE TA TARGE GETS TS AHM AHMDAHL DAHLS LAW AW DIMINISHING RETURNS OBSERVED IF NOT DIFFICULT TO COME UP WITH COMPLEX EXPLICITLY DESIGNED FOR COMBINED TARGETS FOR PSR REQUIREMENTS 5
9/29/19 OPEN SOURCE vs PROPRIETARY HIGH LEVEL PSR RELEVANT DIFFERENCES RE READ: FRE REE !!! OPEN SOURCE TOOLS OFTEN REMOVE THE FIRST BARRIER TO ENTRY, THE COST SU SUPPORT PAID FOR SUPPORT CAN BE CRITICAL AT CRUNCH TIME FIX YOUR OWN ISSUES FI HAVING ACCESS TO THE SOURCE CODE MEANS CUSTOM FEATURES CAN BE ADDED CA CAPABILITY HISTORICALLY, COMMERCIAL TOOLS HAVE HAD MORE BELLS AND WHISTLES CO CONTROL ABILITY TO DEPLOY ON PREM PROVIDES MORE COTROL OVER DATA THE WEB EXPERIENCE END USER PERSPECTIVE 6
9/29/19 THE WEB REQUEST LIFECYCLE THE COMPONENTS THE MAKE UP END USER PERCIEVED RESPONSE TIMES • BROWSER • OS DNS CACHE CHECKS • ROUTER & CONNECTION • ISP • REQUEST SENT OVER TCP SERVER • APP SHELL HTML RENDERS • BASIC HTML PAGE 7
9/29/19 DOM CSSOM RENDER TREE TREE TREE BUILDING THE TREES How the browser begins the process of showing the web page RENDER BLOCKS DOWNLOAD ADDITIONAL RESOURCES CSS, JS, WOFF files may block the parse and/or render trees from being generated 8
9/29/19 AJAX REQUESTS Data is increasingly being populated after the app shell has bootstrapped VISUALLY COMPLETE AND INTERACTIVE Page is fully loaded and is usable. 9
9/29/19 FRONTEND PERFORMANCE METRICS Each of these have an impact on the perception of performance by the user. FIRST PAINT HERO ELEMENT When the app shows some sign of When the most important part of the page life/activty upon submitting request. that the user is interested in has loaded. FIRST CONTENTFUL PAINT TIME TO INTERACTIVE (FIRST) When the canvas of the app has taken the When most of the elements on the page shape and structure of the final form. respond to input from the user. FIRST MEANINGFUL PAINT CONSISTENTLY INTERACTIVE When the user can start to get meaningful When the app is no longer issuing network cues from the app. requests (except websockets) MOBILE vs DESKTOP HIGH LEVEL PSR RELEVANT DIFFERENCES RESOURCES 100 VS 100 MOBILE USERS TEND TO ARE 100 MOBILE USE MORE RESOURCES USERS EQUIVALENT TO ON THE BACKEND 100 DESKTOP USERS? MOBILE EMULATION LICENSES HAVE BEEN MOBILE USERS TEND MORE EXPENSIVE TO HAVE SHORTER LICENSES ATTENTION SPANS EXPECTATIONS 10
Recommend
More recommend
