Cloud WorkBench A Web-Based Framework for Benchmarking Cloud - PowerPoint PPT Presentation

Cloud WorkBench A Web-Based Framework for Benchmarking Cloud Services Joel Scheuner University of Zurich, Switzerland software evolution & architecture lab 14.08.2014

Cloud Computing - Essential Characteristics 1. On-demand Self-service 2. Rapid Elasticity 3. Utility-based Pricing 4. Resource Pooling 5. Broad Network Access 2

Cloud Computing - Essential Characteristics 1. On-demand Self-service 2. Rapid Elasticity 3. Utility-based Pricing 70x 4. Resource Pooling 5. Broad Network Access 3 days 3

Cloud Computing - Essential Characteristics 1. On-demand Self-service 2. Rapid Elasticity 3. Utility-based Pricing 4. Resource Pooling 5. Broad Network Access 4

Cloud Computing - Essential Characteristics 1. On-demand Self-service 2. Rapid Elasticity 3. Utility-based Pricing 4. Resource Pooling 5. Broad Network Access 5

Cloud Computing - Essential Characteristics 1. On-demand Self-service 2. Rapid Elasticity 3. Utility-based Pricing 4. Resource Pooling 5. Broad Network Access 6

Infrastructure-as-a- Service (IaaS) • Processing, storage, networks • Virtual Machines (VMs) >23 Instance Types 7

Differences between IaaS Services Even for services • Performance with the same • Hardware being served specification! • Reliability • Costs

Benchmark • Performance test • Types • Micro-Benchmarks • Application Benchmarks 9

Demo 10

Cloud WorkBench Open Source https://github.com/sealuzh/cloud-workbench 11

Research Questions How can common IaaS cloud benchmarks from I literature be defined in a modular and portable manner? How can benchmarks from RQ1 be periodically II scheduled and reproducibly executed in cloud environments without manual interaction? 12

Overall Architecture CWB Server Access N Web Interface Su REST Web Interface REST Business Logic VM Environment Manager Scheduler Provider Plugin Pro Core Experimenter Relational Database 13

Overall Architecture CWB Server IaaS Providers Access Notify State + IaaS Provider IaaS Provider Web Interface Submit Results REST Web Interface REST CWB Client Library Business Logic Provider API Execution Environment REST VM Environment Benchmark Manager Scheduler Manage VMs Provider Plugin Provision VMs + SSH Core Cloud VM Cloud VMs Execute Commands Experimenter Relational Database Fetch Configuration 14

Overall Architecture CWB Server IaaS Providers Access Notify State + IaaS Provider IaaS Provider Web Interface Submit Results REST Web Interface REST CWB Client Library Business Logic Provider API Execution Environment REST VM Environment Benchmark Manager Scheduler Manage VMs Provider Plugin Provision VMs + SSH Core Cloud VM Cloud VMs Execute Commands Experimenter Relational Database Fetch Configuration REST REST Configurations Upload Provisioning Service Configuration 15

Benchmark Anatomy Timeout 1 * * Benchmark Definition 1 * 1 Schedule 16

Benchmark Anatomy Timeout 1 1..* * 1..* Cloud VM Provisioning * Configuration Configuration * Benchmark Definition 1 * * 1 <<enum>> Result Model 1..* Result Type 1 Schedule 17

Benchmark Execution 18

Case Study • Raw sequential write speed • HDD vs. SSD storage • Different instance types 19

Questions 1 When do larger instance types perform better than smaller instance types? 2 When should larger instance types be preferred over the better block storage type? 3 How do instance types and block storage types influence performance variability? 20

Experiment Setup • Amazon EC2 Ireland (eu-west-1) • Ubuntu 14.04 (trusty) • FIO benchmark Instance Type Price per Hour t1.micro $ 0.020 • 20. - 23. June 2014 m1.small $ 0.047 • 8 - 12 repetitions m3.medium $ 0.077 21

1 When do larger instance types perform better than smaller instance types? 7000 (Elastic Block Storage) Standard EBS General Purpose EBS (SSD) 6000 6000 5500 5000 4000 KB/s 3500 3000 3000 2000 1000 1000 750 0 t1.micro m1.small m3.medium + 4x + 0x 22

2 When should larger instance types be preferred over the better block storage type? 7000 (Elastic Block Storage) Standard EBS +100% General Purpose EBS (SSD) +60% 6000 6000 5500 5000 4000 KB/s 3500 3000 3000 2000 +30% 1000 1000 750 0 t1.micro m1.small m3.medium Larger Instance Type Better Block Storage Type 23

3 How do instance types and block storage types influence performance variability? m1.small + General Purpose EBS m1.small + Standard EBS t1.micro + General Purpose EBS t1.micro + Standard EBS 6000 4000 KB/s 2000 0 0 5 10 15 20 24 min

3 How do instance types and block storage types influence performance variability? s in % of x ̄ t1.micro m1.small m3.medium 20% (20-50%) 20% (10-20%) 30% (15-60%) Standard EBS General 10 % (20-40%) 10% (5-15%) 10% (5-10%) Purpose EBS m1.small + General Purpose EBS m1.small + Standard EBS t1.micro + General Purpose EBS t1.micro + Standard EBS 6000 4000 KB/s 2000 0 0 5 10 15 20 25 min

Conclusion I How can common IaaS cloud benchmarks from I literature be defined in a modular and portable manner? • Entirely define benchmarks by means of code • Apply common software engineering techniques • Make components and benchmarks configurable 26

Conclusion II How can benchmarks from RQ1 be periodically II scheduled and reproducibly executed in cloud environments without manual interaction? • Use system utilities and existing tools to build a fully automated benchmark execution environment • Eliminate any error-prone human interactions threatening reproducibility 27

Outlook • Large-scale evaluation for a 
 World Wide Web Conference paper • Extend CWB in a master project • Web-based benchmarking studio • Support entire lifecycle