CS147 2015-06-15 CS 147: Computer Systems Performance Analysis Workload Selection CS 147: Computer Systems Performance Analysis Workload Selection 1 / 39
Overview CS147 Overview 2015-06-15 Workload Types What is a Workload? Instruction Workloads Synthetic Workloads Standard Benchmarks Exercisers and Drivers Overview Workload Selection Considerations Example Workload Types What is a Workload? Instruction Workloads Synthetic Workloads Standard Benchmarks Exercisers and Drivers Workload Selection Considerations Example 2 / 39
Workload Types What is a Workload? What is a Workload? CS147 What is a Workload? 2015-06-15 Workload Types ◮ Workload : anything a computer is asked to do What is a Workload? ◮ Test workload: any workload used to analyze performance ◮ Real workload: any observed during normal operations ◮ Synthetic workload: created for controlled testing What is a Workload? ◮ Workload : anything a computer is asked to do ◮ Test workload: any workload used to analyze performance ◮ Real workload: any observed during normal operations ◮ Synthetic workload: created for controlled testing 3 / 39
Workload Types What is a Workload? Real Workloads CS147 Real Workloads 2015-06-15 Workload Types ◮ Advantage: represent reality ◮ Disadvantage: uncontrolled What is a Workload? ◮ Can’t be repeated ◮ Can’t be described simply ◮ Difficult to analyze ◮ Nevertheless, often useful for “final analysis” papers Real Workloads ◮ E.g., “We ran system foo and it works well” ◮ Advantage: represent reality ◮ Disadvantage: uncontrolled ◮ Can’t be repeated ◮ Can’t be described simply ◮ Difficult to analyze ◮ Nevertheless, often useful for “final analysis” papers ◮ E.g., “We ran system foo and it works well” 4 / 39
Workload Types What is a Workload? Synthetic Workloads CS147 Synthetic Workloads 2015-06-15 Workload Types ◮ Advantages: ◮ Controllable What is a Workload? ◮ Repeatable ◮ Portable to other systems ◮ Easily modified Synthetic Workloads ◮ Disadvantage: can never be sure real world will be the same ◮ Advantages: ◮ Controllable ◮ Repeatable ◮ Portable to other systems ◮ Easily modified ◮ Disadvantage: can never be sure real world will be the same 5 / 39
Workload Types Instruction Workloads Instruction Workloads CS147 Instruction Workloads 2015-06-15 Workload Types ◮ Useful only for CPU performance ◮ But teach useful lessons for other situations ◮ Development over decades Instruction Workloads ◮ “Typical” instruction ( ADD ) ◮ Instruction mix (by frequency of use) ◮ Sensitive to compiler, application, architecture Instruction Workloads ◮ Still used today (GFLOPS) ◮ Processor clock rate ◮ Only valid within processor family ◮ Useful only for CPU performance ◮ But teach useful lessons for other situations ◮ Development over decades ◮ “Typical” instruction ( ADD ) ◮ Instruction mix (by frequency of use) ◮ Sensitive to compiler, application, architecture ◮ Still used today (GFLOPS) ◮ Processor clock rate ◮ Only valid within processor family 6 / 39
Workload Types Instruction Workloads Instruction Workloads (cont’d) CS147 Instruction Workloads (cont’d) 2015-06-15 Workload Types ◮ Modern complexity makes mixes invalid ◮ Pipelining ◮ Data/instruction caching Instruction Workloads ◮ Prefetching ◮ Kernel is inner loop that does useful work: ◮ Sieve, matrix inversion, sort, etc. Instruction Workloads (cont’d) ◮ Ignores setup, I/O, so can be timed by analysis if desired (at least in theory) ◮ Modern complexity makes mixes invalid ◮ Pipelining ◮ Data/instruction caching ◮ Prefetching ◮ Kernel is inner loop that does useful work: ◮ Sieve, matrix inversion, sort, etc. ◮ Ignores setup, I/O, so can be timed by analysis if desired (at least in theory) 7 / 39
Workload Types Synthetic Workloads Synthetic Workloads CS147 Synthetic Workloads 2015-06-15 Workload Types ◮ Complete programs ◮ Designed specifically for measurement Synthetic Workloads ◮ May do real or “fake” work ◮ May be adjustable (parameterized) ◮ Two major classes: ◮ Real-world benchmarks Synthetic Workloads ◮ Purpose-written exercisers Concern is that real-world benchmarks represent only a specific problem. ◮ Complete programs Concern is that exercisers may not stress system the same way as ◮ Designed specifically for measurement real programs (e.g., page faults). ◮ May do real or “fake” work ◮ May be adjustable (parameterized) ◮ Two major classes: ◮ Real-world benchmarks ◮ Purpose-written exercisers 8 / 39
Workload Types Synthetic Workloads Real-World Benchmarks CS147 Real-World Benchmarks 2015-06-15 Workload Types ◮ Pick a representative application ◮ Pick sample data Synthetic Workloads ◮ Run it on system to be tested ◮ Modified Andrew Benchmark, MAB, is a real-world benchmark Real-World Benchmarks ◮ Easy to do, accurate for that sample data ◮ Fails to consider other applications, data ◮ Pick a representative application ◮ Pick sample data ◮ Run it on system to be tested ◮ Modified Andrew Benchmark, MAB, is a real-world benchmark ◮ Easy to do, accurate for that sample data ◮ Fails to consider other applications, data 9 / 39
Workload Types Synthetic Workloads Application Benchmarks CS147 Application Benchmarks 2015-06-15 Workload Types ◮ Variation on real-world benchmarks ◮ Choose most important subset of functions Synthetic Workloads ◮ Write benchmark to test those functions ◮ Tests what computer will be used for ◮ Need to be sure important characteristics aren’t missed Application Benchmarks ◮ Mix of functions must reflect reality ◮ Variation on real-world benchmarks ◮ Choose most important subset of functions ◮ Write benchmark to test those functions ◮ Tests what computer will be used for ◮ Need to be sure important characteristics aren’t missed ◮ Mix of functions must reflect reality 10 / 39
Workload Types Standard Benchmarks “Standard” Benchmarks CS147 “Standard” Benchmarks 2015-06-15 ◮ Often need to compare general-purpose computer systems Workload Types for general-purpose use ◮ E.g., should I buy an AMD or Intel CPU? ◮ Tougher: Mac or PC? ◮ Desire for an easy, comprehensive answer Standard Benchmarks ◮ People writing articles may need to compare tens of machines ◮ Often need to make comparisons over time ◮ Is this year’s PowerPC faster than last year’s Pentium? “Standard” Benchmarks ◮ Probably yes, but by how much? ◮ Don’t want to spend time writing own code ◮ Often need to compare general-purpose computer systems ◮ Could be buggy or not representative ◮ Need to compare against other people’s results ◮ “Standard” benchmarks offer solution for general-purpose use ◮ E.g., should I buy an AMD or Intel CPU? ◮ Tougher: Mac or PC? ◮ Desire for an easy, comprehensive answer ◮ People writing articles may need to compare tens of machines ◮ Often need to make comparisons over time ◮ Is this year’s PowerPC faster than last year’s Pentium? ◮ Probably yes, but by how much? ◮ Don’t want to spend time writing own code ◮ Could be buggy or not representative ◮ Need to compare against other people’s results ◮ “Standard” benchmarks offer solution 11 / 39
Workload Types Standard Benchmarks Popular “Standard” Benchmarks CS147 Popular “Standard” Benchmarks 2015-06-15 ◮ Sieve, 8 queens, etc. Workload Types ◮ Whetstone ◮ Linpack ◮ Dhrystone Standard Benchmarks ◮ Debit/credit ◮ TPC ◮ SPEC Popular “Standard” Benchmarks ◮ MAB ◮ Winstone, webstone, etc. ◮ Sieve, 8 queens, etc. ◮ Postmark, IOzone, FileBench ◮ . . . ◮ Whetstone ◮ Linpack ◮ Dhrystone ◮ Debit/credit ◮ TPC ◮ SPEC ◮ MAB ◮ Winstone, webstone, etc. ◮ Postmark, IOzone, FileBench ◮ . . . 12 / 39
Workload Types Standard Benchmarks Sieve, etc. CS147 Sieve, etc. 2015-06-15 Workload Types ◮ Prime number sieve (Erastothenes) ◮ Nested for loops ◮ Often such small array that it’s silly Standard Benchmarks ◮ 8 queens ◮ Recursive ◮ Many others Sieve, etc. ◮ Generally not representative of real problems ◮ Prime number sieve (Erastothenes) ◮ Nested for loops ◮ Often such small array that it’s silly ◮ 8 queens ◮ Recursive ◮ Many others ◮ Generally not representative of real problems 13 / 39
Workload Types Standard Benchmarks Whetstone CS147 Whetstone 2015-06-15 Workload Types ◮ Dates way back (can compare against 70’s) ◮ Based on real observed instruction frequencies Standard Benchmarks ◮ Entirely synthetic (no useful result) ◮ Modern optimizers may delete code ◮ Mixed data types, but best for floating-point Whetstone ◮ Be careful of incomparable variants! ◮ Dates way back (can compare against 70’s) ◮ Based on real observed instruction frequencies ◮ Entirely synthetic (no useful result) ◮ Modern optimizers may delete code ◮ Mixed data types, but best for floating-point ◮ Be careful of incomparable variants! 14 / 39
Workload Types Standard Benchmarks LINPACK CS147 LINPACK 2015-06-15 Workload Types ◮ Based on real programs and data Standard Benchmarks ◮ Developed by supercomputer users ◮ Great if you’re doing serious numerical computation LINPACK ◮ Based on real programs and data ◮ Developed by supercomputer users ◮ Great if you’re doing serious numerical computation 15 / 39
Recommend
More recommend
