BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments Nikolas Herbst, Andreas Weber, Henning Groenda, Samuel Kounev Dept. of Computer Science, University of Würzburg FZI Research Center, Karlsruhe SEAMS 2015, Firenze, Italy May 18, 2015 http://descartes.tools/bungee
Characteristics of … Rubber Bands Clouds Base Length Performance (1 resource unit) Width/Thickness/Force Quality Criteria / SLOs Contract: ... . Contract: ... . Contract: … Resp. Time Resp. Time Resp. Time < 2 Sec. < 1 Sec. < 0.5 Sec. Strechability Scalability Elasticity Elasticity Price Price $ $$ $$$ $ $$ $$$ 2 N. Herbst BUNGEE: An IaaS Cloud Elasticity Benchmark
Comparing Elastic Behavior of … IaaS Clouds Rubber Bands … Measure elasticity 2 cm independent of time demand supply … performance 2 cm and 4 cm scalability time demand supply 3 N. Herbst BUNGEE: An IaaS Cloud Elasticity Benchmark
Agenda ? § Motivation § Related Work § Benchmark Concept & Implementation § Evaluation & Case Study § Conclusion 4 N. Herbst BUNGEE: An IaaS Cloud Elasticity Benchmark
Motivation Elasticity: § Mayor quality attribute of clouds [Gartner09] § Many strategies exist [Galante12, Jennings14] § Industry § Academia à Benchmark for comparability! “You can’t control what you can’t measure?” (DeMarco) “If you cannot measure it, you cannot improve it” (Lord Kelvin 5 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Related Work § Specialized approaches [ Binning09, Li10, Dory11, Almeida13 ] § Measure technical provisioning time § Measure SLA compliance § Focus on scale up/out § Business perspective [ Weimann11, Folkerts12, Islam12, Moldovan13, Tinnefeld14 ] § What is the financial impact? § Disadvantage: Mix-up of elasticity technique and business model 6 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Cloud System Under Test 7 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Elasticity Benchmarking Concept Analyze System performance of underlying resources & Analysis scaling behavior Benchmark Calibration Measurement Elasticity Evaluation 8 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Analyze System Phase Approach: resource amount § Evaluate system separately at each scale § Find maximal intensity that the system can withstand without f(intensity ) violating SLO (binary search) § Derive demand step function: resourceDemand = f(intensity) max. load intensity Benefit: § Derive resource demand for arbitrary load intensity variations # resources demand intensity f(intensity) time time 9 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Elasticity Benchmarking Concept Analyze System performance of underlying resources & Analysis scaling behavior Benchmark Benchmark Adjust load profile Calibration Calibration Measurement Elasticity Evaluation 10 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Benchmark Calibration Phase Goal: Induce same resource demand on all systems intensity intensity f(intensity) f(intensity) time time … … resources resources time time demand supply demand supply Approach: Adjust load intensity profile to overcome § Different performance of underlying resources § Different scalability 11 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Elasticity Benchmarking Concept Analyze System performance of underlying resources & Analysis scaling behavior Benchmark Adjust load profile Calibration Expose CSUT to varying load & Measurement monitor resource supply & demand Elasticity Evaluation 12 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Measurement Phase § Requirements: Stress SUT in a representative manner § Realistic variability of load intensity § Adaptability of load profiles to suit different domains § Approach: § Open workload model [Schroeder06] http://descartes.tools/limbo § Model load variations with the LIMBO toolkit [SEAMS15Kistowski] Facilitates creation of new load profiles § Derived from existing traces § With desired properties (e.g. seasonal pattern, bursts) § Execute load profile using JMeter A JMeter Timer-Plugin delays requests according to timestamp file created by LIMBO https://github.com/andreaswe/JMeterTimestampTimer 13 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Elasticity Benchmarking Concept Analyze System performance of underlying resources & Analysis scaling behavior Benchmark Adjust load profile Calibration Expose CSUT to varying load & Measurement CloudStack monitor resource supply & demand Evaluate elasticity aspects Elasticity accuracy & timing Evaluation with metrics 14 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Metrics: Accuracy (1/3) [Herbst13] resources O 2 U 3 O 3 U 2 O 1 U 1 T resource demand resource supply accuracy U accuracy O 15 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Same Value – Different Behavior res. demand res. supply resources time System A res. demand res. supply resources time System B 16 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Metrics: Timeshare (2/3) A 1 B 1 A 2 A 3 B 2 B 3 resources T resource demand resource supply timeshare U timeshare O 17 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Metrics: Jitter (3/3) resources resource demand resource supply resources resource demand resource supply jitter 18 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Elasticity Benchmarking Concept Analyze System performance of underlying resources & Analysis scaling behavior Benchmark Adjust load profile Calibration Expose CSUT to varying load & Measurement CloudStack monitor resource supply & demand Evaluate elasticity aspects Elasticity accuracy & timing Evaluation with metrics 19 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
BUNGEE Implementation § Java-based elasticity benchmarking framework § Components § Harness (Benchmark Node) § Cloud-side load generation application (CSUT) § Automates the four benchmarking activities CloudStack System Analysis Benchmark Calibration Measurement Elasticity Evaluation § Currently: Analysis of horizontally scaling clouds based on § CloudStack § AWS § Extensible with respect to § new cloud management software § new resource types Sources soon available at § new metrics http://descartes.tools/bungee 20 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Evaluation & Case Study § Evaluation (private cloud) § Reproducibility of system analysis Err rel < 5%, confidence 95% for first scaling stage § Simplified system analysis Linearity assumption holds for test system § Consistent ranking by metrics Separate evaluation for each metric, min. 4 configurations per metric § Case Study (private & public cloud) § Applicability in real scenario § Different performance of underlying resources § Metric Aggregation 21 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Evaluation: Accuracy U threshold accuarcy U Down [%] [res. units] 55 0.145 65 0.302 75 0.371 85 0.603 accuracy U allows to rank different elastic behaviors on an ordinal scale 22 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Case Study: Configuration F - 1Core F - 1Core quietTime 120s condTrueDur 30s threshUp 65% threshDown 10% accuarcy O accuarcy O accuracy U accuracy U timeshare O timeshare O timeshare U timeshare U jitter jitter elastic elastic violations violations Configuration Configuration [res. units] [res. units] [res. units] [res. units] [%] [%] [%] [%] [adap/min.] [adap/min.] speedup speedup [%] [%] F – 1Core F – 1Core 2.423 2.423 0.067 0.067 66.1 66.1 4.8 4.8 -0.067 -0.067 1.046 1.046 7.6 7.6 23 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Case Study: Config. F - 2Core not adjusted F - 2Core no adjustment quietTime 120s condTrueDur 30s threshUp 65% threshDown 10% accuarcy O accuracy U timeshare O timeshare U jitter elastic violations Configuration [res. units] [res. units] [%] [%] [adap/min.] speedup [%] F – 1Core 2.423 0.067 66.1 4.8 -0.067 1.046 7.6 F – 2Core no adjustment 1.811 0.001 63.8 0.1 -0.033 1.291 2.1 24 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Case Study: Config. F - 2Core adjusted F - 2Core adjusted quietTime 120s condTrueDur 30s threshUp 65% threshDown 10% accuarcy O accuracy U timeshare O timeshare U jitter elastic violations Configuration [res. units] [res. units] [%] [%] [adap/min.] speedup [%] F – 1Core 2.423 0.067 66.1 4.8 -0.067 1.046 7.6 F – 2Core no adjustment 1.811 0.001 63.8 0.1 -0.033 1.291 2.1 F – 2Core adjusted 2.508 0.061 67.1 4.5 -0.044 1.025 8.2 25 BUNGEE: An IaaS Cloud Elasticity Benchmark N. Herbst
Recommend
More recommend
