CS147 2015-06-15 CS 147: Computer Systems Performance Analysis Ratio Games and Introduction to Experimental Design CS 147: Computer Systems Performance Analysis Ratio Games and Introduction to Experimental Design 1 / 39
Overview CS147 Overview 2015-06-15 Ratio Games How to Lie Strategies for Winning Fair Analysis Overview Experimental Design Introduction 2 k Designs Ratio Games How to Lie Strategies for Winning Fair Analysis Experimental Design Introduction 2 k Designs 2 / 39
Ratio Games How to Lie Ratio Games CS147 Ratio Games 2015-06-15 Ratio Games ◮ Choosing a base system How to Lie ◮ Using ratio metrics ◮ Relative performance enhancement ◮ Ratio games with percentages Ratio Games ◮ Strategies for winning a ratio game ◮ Correct analysis of ratios ◮ Choosing a base system ◮ Using ratio metrics ◮ Relative performance enhancement ◮ Ratio games with percentages ◮ Strategies for winning a ratio game ◮ Correct analysis of ratios 3 / 39
Ratio Games How to Lie Choosing a Base System CS147 Choosing a Base System 2015-06-15 Ratio Games ◮ Run workloads on two systems How to Lie ◮ Normalize performance to chosen system ◮ Take average of ratios ◮ Presto : you control what’s best Choosing a Base System ◮ Run workloads on two systems ◮ Normalize performance to chosen system ◮ Take average of ratios ◮ Presto : you control what’s best 4 / 39
Ratio Games How to Lie Example of Choosing a Base System CS147 Example of Choosing a Base System 2015-06-15 Ratio Games ◮ (Carefully) selected Ficus results: How to Lie 1 2 1 / 2 2 / 1 cp 231.8 168.6 1.37 0.73 rcp 260.6 338.3 0.77 1.30 Example of Choosing a Base System Mean 246.2 253.45 1.07 1.02 Here, the mean running time on two replicas is worse. But by choosing the appropriate base, I can make a single replica 7% ◮ (Carefully) selected Ficus results: slower, or I can make two replicas 2% slower (i.e., a single replica is 1 2 1 / 2 2 / 1 2% faster). cp 231.8 168.6 1.37 0.73 rcp 260.6 338.3 0.77 1.30 Mean 246.2 253.45 1.07 1.02 5 / 39
Ratio Games How to Lie Why Does This Work? CS147 Why Does This Work? 2015-06-15 Ratio Games ◮ Expand the arithmetic: R a ; b = y a R b ; b = 1 . 0 How to Lie y b 1 R i ; a ; b = 1 � y 0 ; a + y 1 ; a � P a ; b = � + · · · Why Does This Work? n n y 0 ; b y 1 ; b � y i ; a 1 1 � = n � = � y 1 ; b 1 P b ; a n ◮ Expand the arithmetic: R is the performance ratio of the overall test, i.e., the total time of all tests (equivalently, their average, assuming paired tests). P is the R a ; b = y a R b ; b = 1 . 0 average of ratios. y b 1 R i ; a ; b = 1 � y 0 ; a � + y 1 ; a � P a ; b = + · · · n n y 0 ; b y 1 ; b � y i ; a 1 1 n � = � = � y 1 ; b 1 P b ; a n 6 / 39
Ratio Games How to Lie Using Ratio Metrics CS147 Using Ratio Metrics 2015-06-15 Ratio Games How to Lie ◮ Pick a metric that is itself a ratio ◮ E.g., power = throughput ÷ response time ◮ Or cost/performance ◮ Handy because division is “hidden” Using Ratio Metrics This is subtler because of the hidden division. ◮ Pick a metric that is itself a ratio ◮ E.g., power = throughput ÷ response time ◮ Or cost/performance ◮ Handy because division is “hidden” 7 / 39
Ratio Games How to Lie Relative Performance Enhancement CS147 Relative Performance Enhancement 2015-06-15 Ratio Games ◮ Compare systems with incomparable bases ◮ Turn into ratios ◮ Example: compare Ficus 1 vs. 2 replicas with UFS vs. NFS (1 How to Lie run on chosen day): “cp” Time Ratio Relative Performance Enhancement Ficus 1 vs. 2 197.4 246.6 1.25 UFS vs. NFS 178.7 238.3 1.33 ◮ “Proves” adding Ficus replica costs less than going from UFS to NFS ◮ Compare systems with incomparable bases ◮ Turn into ratios ◮ Example: compare Ficus 1 vs. 2 replicas with UFS vs. NFS (1 run on chosen day): “cp” Time Ratio Ficus 1 vs. 2 197.4 246.6 1.25 UFS vs. NFS 178.7 238.3 1.33 ◮ “Proves” adding Ficus replica costs less than going from UFS to NFS 8 / 39
Ratio Games How to Lie Ratio Games with Percentages CS147 Ratio Games with Percentages 2015-06-15 Ratio Games ◮ Percentages are inherently ratios ◮ But disguised ◮ So great for ratio games How to Lie ◮ Example: Passing tests Test A Runs A Passes A % B Runs B Passes B % 1 300 60 20 32 8 25 Ratio Games with Percentages 2 50 2 4 500 40 8 Total 350 62 18 532 48 9 ◮ A is worse, but looks better in total line! ◮ Percentages are inherently ratios ◮ But disguised ◮ So great for ratio games ◮ Example: Passing tests Test A Runs A Passes A % B Runs B Passes B % 1 300 60 20 32 8 25 2 50 2 4 500 40 8 Total 350 62 18 532 48 9 ◮ A is worse, but looks better in total line! 9 / 39
Ratio Games How to Lie More on Percentages CS147 More on Percentages 2015-06-15 Ratio Games ◮ Psychological impact ◮ 1000% sounds bigger than 10-fold (or 11-fold) ◮ Great when both original and final performance are lousy How to Lie ◮ E.g., salary went from $40 to $80 per week ◮ Small sample sizes can generate big lies ◮ “83% of dentists surveyed recommend Crest” More on Percentages ◮ (We asked 6 dentists; 5 liked Crest) ◮ Base should be initial, not final value ◮ E.g., price can’t drop 400% ◮ Psychological impact ◮ 1000% sounds bigger than 10-fold (or 11-fold) ◮ Great when both original and final performance are lousy ◮ E.g., salary went from $40 to $80 per week ◮ Small sample sizes can generate big lies ◮ “83% of dentists surveyed recommend Crest” ◮ (We asked 6 dentists; 5 liked Crest) ◮ Base should be initial, not final value ◮ E.g., price can’t drop 400% 10 / 39
Ratio Games Strategies for Winning Can You Win the Ratio Game? CS147 Can You Win the Ratio Game? 2015-06-15 Ratio Games ◮ If one system is better by all measures, a ratio game won’t work Strategies for Winning ◮ But recall percent-passes example ◮ And selecting the base lets you change the magnitude of the difference ◮ If each system wins on some measures, ratio games might be Can You Win the Ratio Game? possible (but no promises) ◮ May have to try all bases ◮ If one system is better by all measures, a ratio game won’t work ◮ But recall percent-passes example ◮ And selecting the base lets you change the magnitude of the difference ◮ If each system wins on some measures, ratio games might be possible (but no promises) ◮ May have to try all bases 11 / 39
Ratio Games Strategies for Winning How to Win Your Ratio Game CS147 How to Win Your Ratio Game 2015-06-15 Ratio Games ◮ For LB metrics, use your system as the base Strategies for Winning ◮ For HB metrics, use the other as a base ◮ If possible, adjust lengths of benchmarks ◮ Elongate when your system performs best ◮ Short when your system is worst How to Win Your Ratio Game ◮ This gives greater weight to your strengths ◮ For LB metrics, use your system as the base ◮ For HB metrics, use the other as a base ◮ If possible, adjust lengths of benchmarks ◮ Elongate when your system performs best ◮ Short when your system is worst ◮ This gives greater weight to your strengths 12 / 39
Ratio Games Fair Analysis Correct Analysis of Ratios CS147 Correct Analysis of Ratios 2015-06-15 Ratio Games Fair Analysis ◮ Previously covered in lecture #5 ◮ Generally, harmonic or geometric means are appropriate ◮ Or use only the raw data Correct Analysis of Ratios ◮ Previously covered in lecture #5 ◮ Generally, harmonic or geometric means are appropriate ◮ Or use only the raw data 13 / 39
Experimental Design Introduction Introduction To Experimental Design CS147 Introduction To Experimental Design 2015-06-15 Experimental Design ◮ You know your metrics ◮ You know your factors Introduction ◮ You know your levels ◮ You’ve got your instrumentation and test loads Introduction To Experimental Design ◮ Now what? ◮ You know your metrics ◮ You know your factors ◮ You know your levels ◮ You’ve got your instrumentation and test loads ◮ Now what? 14 / 39
Experimental Design Introduction Goals in Experiment Design CS147 Goals in Experiment Design 2015-06-15 Experimental Design ◮ Obtain maximum information with minimum work ◮ Typically meaning minimum number of experiments Introduction ◮ More experiments aren’t better if you’re the one who has to perform them ◮ Well-designed experiments are also easier to analyze Goals in Experiment Design ◮ Obtain maximum information with minimum work ◮ Typically meaning minimum number of experiments ◮ More experiments aren’t better if you’re the one who has to perform them ◮ Well-designed experiments are also easier to analyze 15 / 39
Experimental Design Introduction Experimental Replications CS147 Experimental Replications 2015-06-15 Experimental Design ◮ System under study will be run with varying levels of different factors, potentially with differing workloads Introduction ◮ Run with particular set of levels and other inputs is a replication ◮ Often, need to do multiple replications with each set of levels Experimental Replications and other inputs ◮ Usually necessary for statistical validation ◮ System under study will be run with varying levels of different factors, potentially with differing workloads ◮ Run with particular set of levels and other inputs is a replication ◮ Often, need to do multiple replications with each set of levels and other inputs ◮ Usually necessary for statistical validation 16 / 39
Recommend
More recommend
