Betting on Performance a note on hypothesis-driven performance testing James Lewis : @boicy
2
How do we usually make decisions about architecture, design and performance? based on experience based on gut feel and often times up front 3
Can we make design decisions more systematically ? and what would that look like? 4
History The lawful good product owners of the publishing house had long lived in awe and fear of their publishing systems. link hello In awe, for they had made a tremendous amount of Gold, but in fear of the time taken to change them, their slowness and their fragility. A messenger was sent to fetch help from a distant land famed for it’s close mighty wizards. You have taken up the challenge… link to close 7
1. You must save the product owners by rebuilding their website. You start o ff the project. In the course of discussions you discover that your ? goals are three fold: link hello 1. improve availability 2. improve performance 3. reduce the cost of delay An Enterprise Architect approaches and addresses you. close You may use: Summon Walking Skeleton turn to 4 Analysis Paralysis turn to 3 If you have none of these you will have to draw your sword and fi ght (turn to 178 ) link to close 8
1. You must save the product owners by rebuilding their website. You start o ff the project. In the course of discussions you discover that your ? goals are three fold: link hello 1. improve availability 2. improve performance 3. reduce the cost of delay An Enterprise Architect approaches and addresses you. close You may use: Summon Walking Skeleton turn to 4 Analysis Paralysis turn to 3 If you have none of these you will have to draw your sword and fi ght (turn to 178 ) link to close 9
3. You cast Analysis Paralysis at the Enterprise Architect. “Foolish young adventurer” says the link hello architect, “we follow the evolutionary school of architecture and we shall have none of the lawful-evil ways of waterfall”. The last thing you see before everything goes dark is the architect incanting in a strange voice. close You have died. Turn to page 1. link to close 10
1. You must save the product owners by rebuilding their website. You start o ff the project. In the course of discussions you discover that your ? goals are three fold: link hello 1. improve availability 2. improve performance 3. reduce the cost of delay An Enterprise Architect approaches and addresses you. close You may use: Summon Walking Skeleton turn to 4 Analysis Paralysis turn to 3 If you have none of these you will have to draw your sword and fi ght (turn to 178 ) link to close 11
4. Your walking skeleton coalesces in a cloud of noxious gasses and solidi fi es as a java dropwizard application. link hello You reach into your backpack and deploy the content store. Your walking skeleton reaches out it’s skeletal arms and grabs armfuls of raw xml. Would you like to: close S3 Transform the xml inside turn to 6 the skeleton Use a magic box turn to 5 link to close 12
distributed monolithic MVC system app. All logic, transformations and rendering happen in process create another Transform service to content within transform the walking content skeleton 13
So how do you decide which path to follow ? 14
Betting on Performance 15
Betting on Performance 16
Evolutionary Architecture Up front design “ bets allows us to place small the house ”. It collapses bets and then re- the options by evaluate our decisions presupposing the based on outcomes answer (and that there is an answer) Option Option Option Option Option Option $$$$$$$$$ $ $ $ $ $ 17
back to the story 18
5. You throw the magic box in between the walking skeleton and the content store. A villager approaches and exclaims: link hello “this beautiful content I see in front of me seems to take an awful long time to get here” You must somehow make the content content arrive faster. If you have a http cache in your close inventory, you may use it now. S3 Cache in between S3 and content turn to 10 Cache in between skeleton and content turn to 33 link to close 19
ESI / Templating Computationally other service other service expensive service Xslt XML in S3 20
0.8 seconds for first byte 1.5 seconds for page load 21
40 35 Page Load (sec) 30 0.8 seconds for first byte 20 1.5 seconds for page load 10 0 0 0 1 Initial observations 22
ESI / Templating Computationally other service other service expensive service Xslt XML in S3 23
We’ve detected performance problems 24
lets just add a cache 25
ESI / Templating Computationally other service other service expensive service Xslt XML in S3 26
ESI / Templating Computationally other service other service expensive service Xslt XML in S3 27
Cache misses predominate Cache hits predominate Page Load (secs) Page Load (secs) 28
There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton Bonus Joke There are only two hard things in messaging: 2. Exactly Once Delivery 1. Delivery Order 2. Exactly Once Delivery 29
lets just add a cache said no one ever 30
Betting on Performance a note on hypothesis-driven performance testing 31
a note on hypothesis-driven performance testing 32
a note on hypothesis-driven performance testing “In general, we look for a new law by the following process: First we guess it ; then we compute the consequences of the guess to see what would be implied if this law that we guessed is right; then we compare the result of the computation to nature , with experiment or experience, compare it directly with observation, to see if it works. If it disagrees with experiment, it is wrong . In that simple statement is the key to science . It does not make any difference how beautiful your guess is, it does not make any difference how smart you are, who made the guess, or what his name is — if it disagrees with experiment, it is wrong” 33
Observe nature Make a guess Compute implications Compare with Nature Draw conclusion
Observe metrics Observe nature Guess Make a guess Make a small change Compute implications Measure results Compare with Nature Was I right? Draw conclusion
Observable systems Observe nature A brain Make a guess Ability to quickly deploy small changes Compute implications Lightweight probes Compare with Nature Draw conclusion
practices that enable small bets* *not exhaustive 37
Good Monitoring 38
What do we mean when we say “monitoring”? 39
What do we mean when we say “monitoring”? Instrumentation Our code and hardware describes what it’s doing Telemetry We have a way of gathering the data from our services Visualisation We can meaningfully visualise our systems behaviour Alerting We can take action based on this information Predictive alerting We can predict in advance when something may go wrong credit: Dan North
request latency request tracing Service Service health downstream dependencies OS metrics 41
Codahale Metrics Breaker Box Dropwizard Yammer Tenacity Circuit Breakers 42
43
44
Continuous Delivery Safely and sustainably reduce lead time to value 45
46
47
Observe nature Make a guess Compute implications Compare with Nature Draw conclusion 48
49
50
51
ApacheBench ab -n 100 -c 10 “http://sciencesite.com/articles/9dfa1ae0-6ba5-4f2f-…” Siege Vegeta echo "GET http://sciencesite.com/“ | vegeta attack -duration=60s -rate=10 52
Cloud native infrastructure as code 53
54
boto
Good monitoring Observe nature A brain Make a guess Ability to quickly deploy small changes Compute implications Lightweight probes Compare with Nature Draw conclusion
Putting it all together 57
0.8 seconds for first byte 1.5 seconds for page load 40 35 Page Load (sec) 30 20 10 0 0 0 1 Initial observations 58
59
Option: Option: Option: Add CPU Cache S3 Cache content Service content 60
Observe nature XSLT can be expensive - maybe we are CPU bound Make a guess Increasing the # of CPU’s could help Compute implications Compare with Nature Draw conclusion 61
40 35 Page load (sec) 30 20 10 6 0 0 0 1 2 After adding more CPU’s 62
63
Second Guess : Moving compute to the data 64
40 35 Page load (sec) 30 20 10 6 4 0 0 0 1 2 3 Move closer to the data 65
Third Guess: Transformations are slow and could be optimised 66
40 35 Page load (sec) 30 20 10 6 4 3.5 0 0 0 1 2 3 4 Tune xslt transforms 67
Final Guess : And, cache 68
ESI / Templating Computationally other service other service expensive service Xslt XML in S3 69
40 35 Page load (sec) 30 20 10 6 4 3.5 0.2 0 0 0 1 2 3 4 5 After adding a cache 70
71
Recommend
More recommend
