Benchmark suites to measure computer performance Outline Benchmark suites to measure � Motivation computer performance � Benchmarking overview � Generic benchmarks � TPC benchmarks � SPEC benchmarks ICCA’04 � SPEC CPU2000 benchmark Departamento de Informática Universidade do Minho – 22/01/04 � Conclusions Hernâni Correia ICCA’04; 22 nd January 2004 Hernâni Correia Benchmark suites to measure computer performance Benchmark suites to measure computer performance Motivation Benchmarking overview By nature everyone wants the best and the Comparing different design alternatives: fastest machine … � Speed ExecutionT imeY = n � What is meant with “fastest machine?” ExecutionT imeX � How is “speed” measured? Key measurement is time ! � Performance � The same machine is the fastest under all 1 circumstances? ExecutionT imeY Performanc eY Performanc eX = = = n 1 ExecutionT imeX Performanc eY Performanc eX ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia
Benchmark suites to measure computer performance Benchmark suites to measure computer performance Benchmarking overview Benchmarking overview Measuring execution time Benchmarking can be defined has a task of measuring ☺ Processor speed the performance of a system/subsystem (or application) on a task or on a well defined set of tasks. � Disk and memory accesses � Amount and type of memory The task/workload that is measured is the so called � Operating system benchmark. � Compilers � Level of optimization The user’s own workload on it’s own system is � … the best benchmark! ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia Benchmark suites to measure computer performance Benchmark suites to measure computer performance Benchmarking overview Benchmarking overview Types of benchmarks: Benchmarking: what’s important? � Real programs The precise hardware configuration: type of processor, clock speed, � number of CPUs, memory size, cache size, video processor and memory, bus, disk speed, and so on. � Kernels The operating system environment like OS version, filesystem, number � of concurrent users, etc. � Toy benchmarks The version of the benchmark used. � The program language used (same program could have different � execution times if implemented in different languages). � Synthetic benchmarks Compiler used and optimizing level during compilation of benchmarks. � ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia
Benchmark suites to measure computer performance Benchmark suites to measure computer performance Generic benchmarks Generic benchmarks � MIPS � Lynpack benchmark MIPS (or Million Instructions per second) has been one alternative to Derived from a real application which was originated as a collection • metrics that use only time. of linear algebra subroutines in Fortran. It tests floating point MIPS specify the instruction execution rate but it’s dependent on the performance and results are presented in Mflops (millions of floating point instructions per second). instruction set, making difficult to compare MIPS of different machines with different instruction sets. It provides three separate benchmarks: Fortran n = 100, Linpack • n=1000, HPL (Highly Parallel Computing). � Whetstone and Dhrystone benchmarks HPL is a software package that generates and solves a random • Whetstone benchmark was the first intentionally written to measure dense linear system of equations on distributed-memory computers computer performance and was designed to simulate floating point using 64-bit floating point arithmetic. numerical applications. Dhrystone is a benchmark program written for testing system’s integer HPL is the benchmark used for the Top500 report. • performance. ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia Benchmark suites to measure computer performance Benchmark suites to measure computer performance TPC benchmarks TPC benchmarks � TPC (Transaction Processing Performance � TPC-D: database benchmark, whose intent is to simulate ad hoc Council) - consortium of vendors that defines queries characterizing the performance of decision support benchmarks for transaction processing and systems. database domains. � TPC-W is a transactional web benchmark. � Define how the tests should run, how system price � Workload on the system is performed in a controlled internet should be measured and how the results should be commerce environment. � Simulates the activities of a business oriented transactional reported. web server. ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia
Benchmark suites to measure computer performance Benchmark suites to measure computer performance SPEC benchmarks SPEC benchmarks � SPEC (Standard Performance Evaluation � SPECmail2001 – is a mail server benchmark Corporation) - nonprofit consortium made up by (based on Internet SMTP and POP3 hardware/software vendors, universities, customers protocols). and different consultants. � SPECweb99 – is the benchmark for � Intention: develop technically credible and objective system-level benchmarks. measuring performance of web servers. � Benchmarks are derived from real programs � SPECjvm98 – measures performance of Java � placing on the system real workloads. Virtual Machines � producing realistic results. ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia Benchmark suites to measure computer performance Benchmark suites to measure computer performance SPEC CPU2000 benchmark SPEC CPU2000 benchmark � CPU2000 benchmark replace CPU95 in measuring CINT2000 the performance of the computer’s processor (CPU), memory architecture and compilers on the tested system. � Reference machine: Sun Ultra10 workstation with a 300 MHz SPARC processor and 256 MB of memory. � It has two different components: CINT2000 and CFP2000. ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia
Benchmark suites to measure computer performance Benchmark suites to measure computer performance SPEC CPU2000 benchmark SPEC CPU2000 benchmark � Metrics CFP2000 � SPECint2000 / SPECfp2000 speed / normalized ratio � SPECint_base2000 / SPECfp_base2000 � SPECint_rate2000 / SPECfp_rate2000 normalized throughput � SPECint_rate_base2000 / SPECfp_rate_base2000 ICCA’04; 22 nd January 2004 ICCA’04; 22 nd January 2004 Hernâni Correia Hernâni Correia
Recommend
More recommend
