eSulabSolutions Integrating Automated Scalability - PowerPoint PPT Presentation


 
 
 
 
 
 
 
 eSulabSolutions Integrating Automated Scalability Assessment into DevOps Alberto Avritzer et al. The Eighth International Workshop on Load Testing and Benchmarking of Software Systems (LTB 2020)

Our Team Daniel Menasche Vilc Rufino ‘ André van Hoorn Alberto Avritzer eSulabSolutions, 
 Vincenzo Ferme near Princeton University Henning Schulz Barbara Russo Andrea Janes Avritzer et al.: Automated Scalability Assessment in DevOps 2

About Alberto Avritzer eSulabSolutions • Senior Researcher at Siemens and AT&T Bell Labs for about 24 years • Published over 70 papers in journals, refereed conference proceedings, and book chapters in those areas: (http://dblp.uni-trier.de/pers/hd/a/Avritzer:Alberto) • Resilience Assessment and Evaluation of Computing Systems. Springer 2012 • Handbook of Software Aging and Rejuvenation, World Scientific, May 2020, World Scientific (https://worldscientific.com/worldscibooks/10.1142/11673) • Siemens/MTA: The NY Subway PA/CIS Scalability Assessment • Load Testing and Performance Analysis of AT&T Operations support systems: • Monitoring for Software aging and Rejuvenation (1993) • Performance testing using Markov chain (1995) • Founder of eSulabSolutions ( https://esulabsolutions.godaddysites.com/ ): • Automated scalability assessment in DevOps and micro service architecture Avritzer et al.: Automated Scalability Assessment in DevOps 3

Motivation: Recent Scalability Related Disasters Americans crash the Canadian immigration NYS Labor website crashes 4

Motivation: Influence of Poor Performance on the Success of Businesses Avritzer et al.: Automated Scalability Assessment in DevOpsps 5

Examples of Operational Profile Representations Load Level Time Observed load situations Rel. Freq. Load level Empirical distribution of load situations Avritzer et al.: Automated Scalability Assessment in DevOps 6


 * - Automation a M s - C o Agile o n d i e t »DevOps is .. o r i n g Micro- 
 Chaos Docker Cultural services Engineering Change CD Live Polyglot Fast Pipelines Experi- Feedback mentation Practices … a set of 
 intended to 
 reduce the time between committing a change to a system and the change being placed into normal production , 
 while ensuring .« HIGH QUALITY COST- ELASTIC EFFICIENT RESILIENT RELIABLE FAST SECURE Avritzer et al.: Automated Scalability Assessment in DevOps 7

* - Automation a M s - C o Agile o n d i e t My Questions of Interest o r i n g Micro- 
 Chaos Docker Cultural services Engineering Change CD Live Polyglot Fast Pipelines Experi- Feedback mentation How can 
 DevOps practices support 
 quality assurance? HIGH QUALITY How to integrate „established“ quality assurance with DevOps COST- ELASTIC practices? EFFICIENT RESILIENT RELIABLE FAST SECURE A. van Hoorn (U Stuttgart) – Continuous Engineering of Dependable Distributed Software Systems 
 Avritzer and van Hoorn: Operational Profile Data for Continuous Dependability Assessment in DevOps 8

• PPTAM: Production and Performance Testing Based Application Monitoring Avritzer et al.: Automated Scalability Assessment in DevOps

Scalability Requirement Automated Measurement 10 Avritzer et al.: Automated Scalability Assessment in DevOps

The Classic Load Testing Approach … and Classic Problems Load tests need much time to execute High manual effort for maintaining load tests Expert Load Driver Expert/Basic Rules LOA SUT D Workload Specification Test Results Evaluation Complex analysis of performance regressions There are no suitable load tests Avritzer et al.: Automated Scalability Assessment in DevOps 11


 
 
 Load Testing in Continuous Delivery Pipelines … How Problems Get Worse Load tests need much time to execute vs. High manual effort for maintaining load tests Fast & frequent releases vs. Pipeline automation void main(String[] args) { 
 int foo; 
 
 // do something 
 bar(foo); 
 System.out.println(“Hi“); 
 } void main(String[] args) { 
 int foo; 
 main(Strin 
 
 int foo; 
 // do something 
 
 bar(foo); 
 // do somet 
 bar(foo); System.out.println(“Hi“); 
 } Build Implementation Functional Testing Performance Testing void main(String[] args) { 
 int foo; 
 
 // do something 
 bar(foo); 
 System.out.println(“Hi“); 
 } Complex load tests for every release impossible Service-focus requires multiple vs. tests vs. Complex analysis of performance regressions There are no suitable load tests Avritzer et al.: Automated Scalability Assessment in DevOps 12

• PPTAM: Production and Performance Testing Based Application Monitoring Avritzer et al.: Automated Scalability Assessment in DevOps

Quantitative Assessment of Deployment Alternatives • Challenge: assess performance of architectural deployment alternatives 
 (e.g., number of replicas, CPU/memory allocation, technology stack) under fuzzy requirements • Our approach • Use operational data to generate and weigh load tests • Measure baseline requirements • Metric allows quantitative comparison of deployment alternatives • Builds on previous work from telecommunication systems: 
 Avritzer, A., Weyuker, E.J.: The automatic generation of load test suites and the assessment of the resulting software. IEEE Trans. Softw. Eng. 21(9) (Sep 1995) Alberto Avritzer, Vincenzo Ferme, Andrea Janes, Barbara Russo, Henning Schulz, and André van Hoorn: 
 A Quantitative Approach for the Assessment of Microservice Architecture Deployment Alternatives by Automated Performance Testing . 
 In Proceedings of the 12th European Conference on Software Architecture (ECSA). LNCS, Springer, 2018 (Accepted) 14

Overview of Approach 1 Rel. Freq. Load Level Time Load level Production Observed load situations Empirical distribution of load situations 3 2 Aggr. Rel. Freq. 0.12 0.14 0.20 0.16 0.11 0.73 Test results Scalability criteria Sampled load level Baseline test Empirical Distribution of Load situations 4 Domain Metric Deployment conf. Sampled load tests Avritzer et al.: Automated Scalability Assessment in DevOps 15

Experiments 3 1,2 12 microservices 10 configurations 6 Load Levels CPU RAM 50,100,150,200,250,300 Concurrent user sessions Replicas Production Deployment Config. Empirical Distribution of Load situations Scal = avg + 3 σ Sampled Load Tests Custom Op. Mix Avritzer et al.: Automated Scalability Assessment in DevOps 16

Experiment Results: Computation of Domain Metric (1/2) Users Aggr. Rel. Freq. 50 0.10582 Scalability API 100 0.18519 Max: 0.20370 Criteria 150 0.22222 GET / PASS 200 0.22222 GET /cart PASS Actual: 0.13580 250 0.20370 POST /item FAIL 300 0.06085 Custom Op. Mix Aggr. Rel. Freq. Contrib. to Domain Metric Deployment Configuration: 1 GB RAM, 0.25 CPU, 1 Replica 17 Avritzer et al.: Automated Scalability Assessment in DevOps

Experiment Results: Computation of Domain Metric (2/2) Users Contribution Max: 1 50 0.10582 100 0.18519 150 0.22222 Actual: 200 0.07999 0.77631 250 0.13580 300 0.04729 4 Domain Metric Contrib. to Domain Metric Deployment Configuration: 1 GB RAM, 0.25 CPU, 1 Replica Avritzer et al.: Automated Scalability Assessment in DevOps 18

Experiment Results: Single-Metric Comparison of Alternatives Avritzer et al.: Automated Scalability Assessment in DevOps 19

Experiment Results: Visual Comparison of Alternatives Max Contrib. Contrib. to Domain Metric Depl. Conf. Sampled Load Tests Avritzer et al.: Automated Scalability Assessment in DevOps 20

PPTAM • Production and Performance Testing Based Application Monitoring Avritzer et al.: Automated Scalability Assessment in DevOps

Mirai PPTAM Avritzer and Russo: Operational Profile Data for Continuous Dependability Assessment in DevOps

Mirai BotNet • Mirai is a malware that has been used to turn networked devices running Linux into remotely controlled bots • We use it to attack the system. It can perform different types of attack • by now, we have explored http, syn, ack 23 Avritzer and Russo: Operational Profile Data for Continuous Dependability Assessment in DevOps

PPTAM, No attack and attack plots with 50 users 24

Identify Scalability Impacting Architecture Components • Apply the approach to a large telecom system Mirai Arch Cockpi t APM

Identify Scalability Impacting Architecture Components from Performance Testing Log Counter_name Measurement Load Service_name Response Time_X Y L S Response Time_X Y L S Response Time_X Y L S Response Time_X Y L S What is the problem with this log? 26 Avritzer et al.: Automated Scalability Assessment in DevOps

Compute Scalability Baseline, Normalized Distance from Baseline, and Linear Regression Slope Counter_Name Low Load_ RT Std _RT Baseline_RT Response Time X 2.488602 3.513397 12.033352 27