CS 147: Computer Systems Performance Analysis One-Factor - PowerPoint PPT Presentation

CS147 2015-06-15 CS 147: Computer Systems Performance Analysis One-Factor Experiments CS 147: Computer Systems Performance Analysis One-Factor Experiments 1 / 42

Overview CS147 Overview 2015-06-15 Introduction The Model Finding Effects Calculating Errors ANOVA Allocation Overview Analysis Verifying Assumptions Introduction Unequal Sample Sizes The Model Finding Effects Calculating Errors ANOVA Allocation Analysis Verifying Assumptions Unequal Sample Sizes 2 / 42

Introduction Characteristics of One-Factor Experiments CS147 Characteristics of One-Factor Experiments 2015-06-15 Introduction ◮ Useful if there’s only one important categorical factor with more than two interesting alternatives ◮ Methods reduce to 2 1 factorial designs if only two choices ◮ If single variable isn’t categorical, should use regression instead Characteristics of One-Factor Experiments ◮ Method allows multiple replications ◮ Useful if there’s only one important categorical factor with more than two interesting alternatives ◮ Methods reduce to 2 1 factorial designs if only two choices ◮ If single variable isn’t categorical, should use regression instead ◮ Method allows multiple replications 3 / 42

Introduction Comparing Truly Comparable Options CS147 Comparing Truly Comparable Options 2015-06-15 Introduction ◮ Evaluating single workload on multiple machines ◮ Trying different options for single component ◮ Applying single suite of programs to different compilers Comparing Truly Comparable Options ◮ Evaluating single workload on multiple machines ◮ Trying different options for single component ◮ Applying single suite of programs to different compilers 4 / 42

Introduction When to Avoid It CS147 When to Avoid It 2015-06-15 Introduction ◮ Incomparable “factors” ◮ E.g., measuring vastly different workloads on single system ◮ Numerical factors ◮ Won’t predict any untested levels ◮ Regression usually better choice When to Avoid It ◮ Related entries across level ◮ Use two-factor design instead ◮ Incomparable “factors” ◮ E.g., measuring vastly different workloads on single system ◮ Numerical factors ◮ Won’t predict any untested levels ◮ Regression usually better choice ◮ Related entries across level ◮ Use two-factor design instead 5 / 42

The Model An Example One-Factor Experiment CS147 An Example One-Factor Experiment 2015-06-15 The Model ◮ Choosing authentication server for single-sized messages ◮ Four different servers are available ◮ Performance measured by response time ◮ Lower is better An Example One-Factor Experiment ◮ Choosing authentication server for single-sized messages ◮ Four different servers are available ◮ Performance measured by response time ◮ Lower is better 6 / 42

The Model The One-Factor Model CS147 The One-Factor Model 2015-06-15 The Model ◮ y ij = µ + α j + e ij ◮ y ij is i th response with factor set at level j ◮ µ is mean response ◮ α j is effect of alternative j � α j = 0 The One-Factor Model ◮ e ij is error term � e ij = 0 ◮ y ij = µ + α j + e ij ◮ y ij is i th response with factor set at level j ◮ µ is mean response ◮ α j is effect of alternative j � α j = 0 ◮ e ij is error term � e ij = 0 7 / 42

The Model One-Factor Experiments With Replications CS147 One-Factor Experiments With Replications 2015-06-15 The Model ◮ Initially, assume r replications at each alternative of factor ◮ Assuming a alternatives, we have a total of ar observations ◮ Model is thus r a a r a � � � � � One-Factor Experiments With Replications y ij = ar µ + r α j + e ij i = 1 j = 1 j = 1 i = 1 j = 1 ◮ Initially, assume r replications at each alternative of factor ◮ Assuming a alternatives, we have a total of ar observations ◮ Model is thus r a a r a � � � � � y ij = ar µ + r α j + e ij i = 1 j = 1 j = 1 i = 1 j = 1 8 / 42

The Model Sample Data for Our Example CS147 Sample Data for Our Example 2015-06-15 The Model ◮ Four alternatives, with four replications each (measured in seconds) A B C D 0.96 0.75 1.01 0.93 1.05 1.22 0.89 1.02 Sample Data for Our Example 0.82 1.13 0.94 1.06 0.94 0.98 1.38 1.21 ◮ Four alternatives, with four replications each (measured in seconds) A B C D 0.96 0.75 1.01 0.93 1.05 1.22 0.89 1.02 0.82 1.13 0.94 1.06 0.94 0.98 1.38 1.21 9 / 42

The Model Finding Effects Computing Effects CS147 Computing Effects 2015-06-15 The Model ◮ Need to figure out µ and α j ◮ We have various y ij ’s ◮ Errors should add to zero: Finding Effects r a � � e ij = 0 i = 1 j = 1 Computing Effects ◮ Similarly, effects should add to zero: a � α j = 0 j = 1 ◮ Need to figure out µ and α j ◮ We have various y ij ’s ◮ Errors should add to zero: r a � � e ij = 0 i = 1 j = 1 ◮ Similarly, effects should add to zero: a � α j = 0 j = 1 10 / 42

The Model Finding Effects Calculating µ CS147 Calculating µ 2015-06-15 The Model ◮ By definition, sum of errors and sum of effects are both zero: r a � � y ij = ar µ + 0 + 0 Finding Effects i = 1 j = 1 ◮ And thus, µ is equal to grand mean of all responses Calculating µ r a µ = 1 � � y ij = y ·· ar i = 1 j = 1 ◮ By definition, sum of errors and sum of effects are both zero: r a � � y ij = ar µ + 0 + 0 i = 1 j = 1 ◮ And thus, µ is equal to grand mean of all responses r a µ = 1 � � y ij = y ·· ar i = 1 j = 1 11 / 42

The Model Finding Effects Calculating µ for Our Example CS147 Calculating µ for Our Example 2015-06-15 The Model Thus, 4 4 Finding Effects 1 = � � y ij µ 4 × 4 i = 1 j = 1 1 = 16 × 16 . 29 Calculating µ for Our Example = 1 . 018 Thus, 4 4 1 � � = y ij µ 4 × 4 i = 1 j = 1 1 = 16 × 16 . 29 = 1 . 018 12 / 42

The Model Finding Effects Calculating α j CS147 Calculating α j 2015-06-15 The Model ◮ α j is vector of responses ◮ One for each alternative of the factor ◮ To find vector, find column means Finding Effects r y · j = 1 � y ij r i = 1 Calculating α j ◮ Separate mean for each j ◮ Can calculate directly from observations ◮ α j is vector of responses ◮ One for each alternative of the factor ◮ To find vector, find column means r y · j = 1 � y ij r i = 1 ◮ Separate mean for each j ◮ Can calculate directly from observations 13 / 42

The Model Finding Effects Calculating Column Mean CS147 Calculating Column Mean 2015-06-15 The Model ◮ We know that y ij is defined to be y ij = µ + α j + e ij Finding Effects ◮ So, r 1 y · j = � ( µ + α j + e ij ) r Calculating Column Mean i = 1 � r � 1 � = r µ + r α j + e ij r i = 1 ◮ We know that y ij is defined to be y ij = µ + α j + e ij ◮ So, r 1 � y · j = ( µ + α j + e ij ) r i = 1 � r � 1 � = r µ + r α j + e ij r i = 1 14 / 42

The Model Finding Effects Calculating Parameters CS147 Calculating Parameters 2015-06-15 The Model ◮ Sum of errors for any given row is zero, so 1 y · j = r ( r µ + r α j + 0 ) Finding Effects = µ + α j Calculating Parameters ◮ So we can solve for α j : α j = y · j − µ = y · j − y ·· ◮ Sum of errors for any given row is zero, so 1 = r ( r µ + r α j + 0 ) y · j = µ + α j ◮ So we can solve for α j : α j = y · j − µ = y · j − y ·· 15 / 42

The Model Finding Effects Parameters for Our Example CS147 Parameters for Our Example 2015-06-15 The Model Server A B C D Col. Mean .9425 1.02 1.055 1.055 Finding Effects Subtract µ from column means to get parameters: Parameters for Our Example Parameters -.076 .002 .037 .037 Server A B C D Col. Mean .9425 1.02 1.055 1.055 Subtract µ from column means to get parameters: Parameters -.076 .002 .037 .037 16 / 42

The Model Calculating Errors Estimating Experimental Errors CS147 Estimating Experimental Errors 2015-06-15 The Model ◮ Estimated response is ˆ y ij = µ + α ij ◮ But we measured actual responses Calculating Errors ◮ Multiple responses per alternative ◮ So we can estimate amount of error in estimated response ◮ Use methods similar to those used in other types of Estimating Experimental Errors experiment designs ◮ Estimated response is ˆ y ij = µ + α ij ◮ But we measured actual responses ◮ Multiple responses per alternative ◮ So we can estimate amount of error in estimated response ◮ Use methods similar to those used in other types of experiment designs 17 / 42

The Model Calculating Errors Sum of Squared Errors CS147 Sum of Squared Errors 2015-06-15 The Model ◮ SSE estimates variance of the errors: r a Calculating Errors SSE = � � e 2 ij i = 1 j = 1 ◮ We can calculate SSE directly from model and observations Sum of Squared Errors ◮ Also can find indirectly from its relationship to other error terms ◮ SSE estimates variance of the errors: r a � � e 2 SSE = ij i = 1 j = 1 ◮ We can calculate SSE directly from model and observations ◮ Also can find indirectly from its relationship to other error terms 18 / 42