A Cloud Benchmark Suite Combining Micro and Application Benchmarks - - PowerPoint PPT Presentation

▶
a cloud benchmark suite combining micro and application
SMART_READER_LITE
LIVE PREVIEW

A Cloud Benchmark Suite Combining Micro and Application Benchmarks - - PowerPoint PPT Presentation

Sep 10, 2022 427 likes •662 views

A Cloud Benchmark Suite Combining Micro and Application Benchmarks Joel Scheuner, Philipp Leitner Joel Scheuner scheuner@chalmers.se joe4dev @joe4dev Context: Public Infrastructure-as-a-Service Clouds IaaS PaaS SaaS Applications

slide-1
SLIDE 1

Joel Scheuner scheuner@chalmers.se joe4dev @joe4dev

A Cloud Benchmark Suite Combining Micro and Application Benchmarks

Joel Scheuner, Philipp Leitner

slide-2
SLIDE 2

Context: Public Infrastructure-as-a-Service Clouds

IaaS PaaS SaaS

Applications Data Runtime Middleware OS Virtualization Servers Storage Networking Applications Data Runtime Middleware OS Virtualization Servers Storage Networking Applications Data Runtime Middleware OS Virtualization Servers Storage Networking

User-Managed Provider-Managed

Infrastructure-as-a-Service (IaaS) Platform-as-a-Service (PaaS) Software-as-a-Service (SaaS)

slide-3
SLIDE 3

2018-04-10 QUDOS@ICPE'18 3

Motivation: Capacity Planning in IaaS Clouds

What cloud provider should I choose?

https://www.cloudorado.com

slide-4
SLIDE 4

20 40 60 80 100 120 2 6 2 7 2 8 2 9 2 1 2 1 1 2 1 2 2 1 3 2 1 4 2 1 5 2 1 6 2 1 7 Number of Instance Type

2018-04-10 QUDOS@ICPE'18 4

Motivation: Capacity Planning in IaaS Clouds

What cloud service (i.e., instance type) should I choose?

t2.nano 0.05-1 vCPU 0.5 GB RAM $0.006/h x1e.32xlarge 128 vCPUs 3904 GB RAM $26.688 hourly

slide-5
SLIDE 5

2018-04-10 QUDOS@ICPE'18 5

Topic: Performance Benchmarking in the Cloud

“The instance type itself is a very major tunable parameter”

@brendangregg re:Invent’17 https://youtu.be/89fYOo1V2pA?t=5m4s

slide-6
SLIDE 6

2018-04-10 QUDOS@ICPE'18 6

Background

Generic Artificial Resource- specific Specific Real-World Resource- heterogeneous

Micro Benchmarks

CPU Memory I/O Network Overall performance (e.g., response time)

Application Benchmarks

slide-7
SLIDE 7

2018-04-10 QUDOS@ICPE'18 7

Related Work

Micro Benchmarking / Application Kernels

Iosup et. al. Performance analysis of cloud computing services for many-tasks scientific computing. Ostermann et. al. A performance analysis of EC2 cloud computing services for scientific computing.

Application Benchmarking

Ferdman et. al. Clearing the clouds: a study of emerging scale-out workloads on modern hardware. Cooper et. al. Benchmarking Cloud Serving Systems with YCSB.

Repeatability of Cloud Experiments

Abedi and Brecht. Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments. @ICPE’17

A A A B B B C C C A B C A B C A B C B A C C B A A C B

B) Multiple Consecutive Trials (MCT) C) Multiple Interleaved Trials (MIT) D) Randomized Multiple Interleaved Trials (RMIT) A) Single Trial

A B C

slide-8
SLIDE 8

2018-04-10 QUDOS@ICPE'18 8

Problem: Isolation, Reproducibility of Execution

Generic Artificial Resource-specific Specific Real-World Resource- heterogeneous

Micro Benchmarks

CPU Memory I/O Network Overall performance (e.g., response time)

Application Benchmarks

slide-9
SLIDE 9

2018-04-10 QUDOS@ICPE'18 9

Question:

Generic Artificial Resource-specific Specific Real-World Resource- heterogeneous

Micro Benchmarks

CPU Memory I/O Network Overall performance (e.g., response time)

Application Benchmarks How can we systematically combine and execute micro and application benchmarks?

slide-10
SLIDE 10

2018-04-10 QUDOS@ICPE'18 10

Idea

Generic Artificial Resource-specific Specific Real-World Resource- heterogeneous

Micro Benchmarks

CPU Memory I/O Network Overall performance (e.g., response time)

Application Benchmarks

Systematically Execute Together

slide-11
SLIDE 11

2018-04-10 QUDOS@ICPE'18 11

Execution Methodology

B A C C B A A C B

D) Randomized Multiple Interleaved Trials (RMIT)

30 benchmark scenarios 3 trials ~2-3h runtime

slide-12
SLIDE 12

2018-04-10 QUDOS@ICPE'18 12

Benchmark Manager

Cloud WorkBench (CWB) Tool for scheduling cloud experiments sealuzh/cloud-workbench

CloudCom 2014 “Cloud Work Bench – Infrastructure-as-Code Based Cloud Benchmarking”

Scheuner, Leitner, Cito, and Gall

Demo@WWW 2015

Scheuner, Cito, Leitner, and Gall

slide-13
SLIDE 13

2018-04-10 QUDOS@ICPE'18 13

Architecture Overview

slide-14
SLIDE 14

2018-04-10 QUDOS@ICPE'18 14

CPU

  • sysbench/cpu-single-thread
  • sysbench/cpu-multi-thread
  • stressng/cpu-callfunc
  • stressng/cpu-double
  • stressng/cpu-euler
  • stressng/cpu-ftt
  • stressng/cpu-fibonacci
  • stressng/cpu-int64
  • stressng/cpu-loop
  • stressng/cpu-matrixprod

Memory

  • sysbench/memory-4k-block-size
  • sysbench/memory-1m-block-size

Broad resource coverage and specific resource testing

Micro Benchmarks

Micro Benchmarks

CPU Memory I/O Network I/O

  • [file I/O] sysbench/fileio-1m-seq-write
  • [file I/O] sysbench/fileio-4k-rand-read
  • [disk I/O] fio/4k-seq-write
  • [disk I/O] fio/8k-rand-read

Network

  • iperf/single-thread-bandwidth
  • iperf/multi-thread-bandwidth
  • stressng/network-epoll
  • stressng/network-icmp
  • stressng/network-sockfd
  • stressng/network-udp

Software (OS)

  • sysbench/mutex
  • sysbench/thread-lock-1
  • sysbench/thread-lock-128
slide-15
SLIDE 15

2018-04-10 QUDOS@ICPE'18 15

Micro Benchmarks: Examples

Micro Benchmarks

CPU Memory I/O Network

1) Prepare

I/O

2) Run 3) Extract Result File I/O: 4k random read 4) Cleanup

3.5793 MiB/sec

Network

Bandwidth Server Client Result

972 Mbits/sec

slide-16
SLIDE 16

2018-04-10 QUDOS@ICPE'18 16

Application Benchmarks

Overall performance (e.g., response time)

Application Benchmarks

Molecular Dynamics Simulation (MDSim) WordPress Benchmark (WPBench)

Multiple short blogging session scenarios (read, search, comment)

20 40 60 80 100 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 Elapsed Time [min] Number of Concurrent Threads
slide-17
SLIDE 17

2018-04-10 QUDOS@ICPE'18 17

Performance Data Set

eu + us eu + us eu

m1.small 1 1 1.7 PV Low m1.medium 1 2 3.75 Instance Type vCPU ECU RAM [GiB] Virtualization Network Performance PV Moderate m3.medium 1 3 3.75 PV /HVM Moderate 2 m1.large 4 7.5 PV Moderate 2 m3.large 6.5 7.5 HVM Moderate 2 m4.large 6.5 8.0 HVM Moderate 2 c3.large 7 3.75 HVM Moderate c4.large 2 8 3.75 HVM Moderate 4 c3.xlarge 14 7.5 HVM Moderate 4 c4.xlarge 16 7.5 HVM High c1.xlarge 8 20 7 PV High

* * ECU := Elastic Compute Unit (i.e., Amazon’s metric for CPU performance)

>240 Virtual Machines (VMs) à 3 Iterations à ~750 VM hours >60’000 Measurements (258 per instance)

slide-18
SLIDE 18

2018-04-10 QUDOS@ICPE'18 18

Cost/Performance is a trade-off but there exist unfavorable instance types

WPBench Response Time Cost Frontier

Cost-Optimal Instance Types Frontier

1000 2000 0.2 0.4 0.6

Instance Cost (USD/h) WPBench Scenario Read Response Time (ms) Instance Type

c1.xlarge c3.large c3.xlarge c4.large c4.xlarge m1.large m1.medium m1.small m3.large m3.medium (hvm) m3.medium (pv) m4.large

+40% performance

  • 40% cost
  • 80% performance
  • 35% cost
slide-19
SLIDE 19

2018-04-10 QUDOS@ICPE'18 19

Almost perfect stability in comparison to previous results

Intra-Cloud Network Bandwidth over Time

300 400 500 600 2017−04−04 2017−04−06 2017−04−08 2017−04−10 2017−04−12 2017−04−14 2017−04−16

Time Network Bandwidth (Mbits/sec) Instance Type

m1.small m3.large m3.medium (hvm)

2014

  • P. Leitner, J. Cito. Patterns in the

Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds. TOIT 2016

2017

slide-20
SLIDE 20

2018-04-10 QUDOS@ICPE'18 20

The newer virtualization type hvm is more I/O efficient than pv

Disk Utilization during I/O Benchmark

90 95 98.5 98.7 98.9 99.1

FIO 4k Sequential Write Disk Utilization (%) FIO 8k Random Read Disk Utilization (%) Instance Type

c1.xlarge c3.large c3.xlarge c4.large c4.xlarge m1.large m1.medium m1.small m3.large m3.medium (hvm) m3.medium (pv) m4.large

Virtualization Type

hvm pv

slide-21
SLIDE 21

2018-04-10 QUDOS@ICPE'18 21

Future Work

Benchmark Design Benchmark Execution Data Pre- Processing Data Analysis

  • 4.41
4.3 3.16 3.32 6.83 5 10 20 30 40 50 m1.small (eu) m1.small (us) m3.medium (eu) m3.medium (us) m3.large (eu) Configuration [Instance Type (Region)] Relative Standard Deviation (RSD) [%]

QUDOS@ICPE 2018 “A Cloud Benchmark Suite Combining Micro and Applications Benchmarks”

Scheuner and Leitner

“Estimating Cloud Application Performance Based on Micro Benchmark Profiling”

Scheuner and Leitner

Under Submission Accepted

slide-22
SLIDE 22

2018-04-10 QUDOS@ICPE'18 22

Conclusions

Selecting an optimal instance type can save up to 40% costs while increasing up to 40% performance

scheuner@chalmers.se

Support trend towards more predictable performance (AWS EC2) The newer virtualization type (hvm) improves I/O utilization rates up to 10% (vs pv)