Performance Chall llenges along the Conti tinuous Deli livery - PowerPoint PPT Presentation

Performance Chall llenges along the Conti tinuous Deli livery Pipelin ine Wolfgang Gottesheim @gottesheim 1 #Dynatrace

Example of a “Bad” Web Deployment 282! Objects 9.68MB Page Size on that page 8.8s Page Load Time Most objects are images delivered from your main domain Very long Connect time (1.8s) to your CDN 2 #Dynatrace

Application problems in the wild 526s to render a financial transaction report 1 SQL running 210s! Debug Logging with log4j on outdate ted log4j library (sync issue) 3 #Dynatrace

But now we‘re doing Continuous Delivery 4 #Dynatrace

700 Deployments / Year 10+ Deployments / Day 50-60 Deployments / Day Every 11.6 seconds 5 #Dynatrace

(only) It‘s not about the pipeline… 6 #Dynatrace

Commit Acceptance Capacity UAT Release Stage Testing Testing Developer s 7 #Dynatrace

Metrics to the rescue! 8 #Dynatrace

We want to replace magic with data! Commit Acceptance Capacity UAT Release Stage Testing Testing 9 #Dynatrace

4 examples 10 #Dynatrace

#1 #1 11 #Dynatrace

Mobile Landing Page of Super Bowl Ad Total size of ~ 20MB 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … 12 #Dynatrace

8MB background image for STPCon 13 #Dynatrace

Key Metrics # Resources Size of Resources Page Size 14 #Dynatrace

#2 #2 15 #Dynatrace

#1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory 16 #Dynatrace

#2: On individual connections 12444! individual connections (and no pooling…) Individual SQL really <1ms Classical N+1 Query Problem 17 #Dynatrace

#3: Homegrown Caching Framework Lots of time spent in Hashtable.get Called from their Entity Objects 18 #Dynatrace

Key Metrics # Functional Errors # of SQL Calls # of same SQL Execs (1+N) # of Connections Rows/Data Transferred 19 #Dynatrace

#3 #3 Si Simply spli litting web servic ice call lls is is not t a mig igratio ion to a mic icroservice architecture 20 #Dynatrace

33! Different connections used 3136! Calls to H2 mostly executed on async background threads DB Exceptions on both DB Exceptions on both Databases Databases 40! internal Web Service Calls that do all these DB Updates 21671! Calls to Oracle 21 #Dynatrace

Key Metrics # Service Calls # of Threads Timing (execution, wait) # SQL executions # of SAME SQLs 22 #Dynatrace

#4 #4 23 #Dynatrace

Is this a successful new Build? 24 #Dynatrace

Look at Resource Usage: CPU, Memory, … 25 #Dynatrace

Memory? Look at Heap Generations 26 #Dynatrace

Root Cause: Dependency Injection 27 #Dynatrace

Prevent: Monitor Memory Metrics for every Build 28 #Dynatrace

Key Metrics # of Objects per Generation # of GC Runs Total Impact of GC 29 #Dynatrace

Where to sta tart? 1. # Resources 2. Size of Resources 3. Page Size 4. # Functional Errors 5. # SQL Executions 6. # of SAME SQLs 30 #Dynatrace

Allow me to introduce… ick P gree A easure M ntegrat I e 31 #Dynatrace

AND MANY MORE Kieker 32 #Dynatrace

We successfully replaced magic with data! Commit Acceptance Capacity UAT Release Stage Testing Testing 33 #Dynatrace

Just one more thing … 34 #Dynatrace

It‘s al also about the pipeline…  Define pipeline metrics! 35 #Dynatrace

Connect your Tests with Quality Let’s look behind the scenes Architectural Data Test Framework Results Build # Test Case Status # SQL # Excep CPU Build 17 testPurchase OK 12 0 120ms testSearch OK 3 1 68ms Build 18 testPurchase FAILED 12 5 60ms testSearch OK 3 1 68ms 75 0 230ms Build 19 testPurchase OK 3 1 68ms testSearch OK Build 20 testPurchase OK 12 0 120ms Exceptions probably reason for testSearch OK 3 1 68ms We identified a regresesion failed tests Problem fixed but now we have an Problem fixed but now we have an Problem solved architectural regression architectural regression Now we have the functional and architectural confidence 36 #Dynatrace

#1: Analyzing each Test #3: Detecting Regression based on Measure #2: Metrics for each Test 37 #Dynatrace

Quality-Metrics based Build Status 38 #Dynatrace

Pull data into Jenkins, Bamboo ... 39 #Dynatrace

Make Quality a first-class citizen „not cool enough“ „Too hard“ „we‘ll get round to this later“ 40 #Dynatrace

Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial Twitter @gottesheim wolfgang.gottesheim@dynatrace.com 41 @Dynatrace