CS 147: Computer Systems Performance Analysis Two-Factor Designs 1 - PowerPoint PPT Presentation

CS147 2015-06-15 CS 147: Computer Systems Performance Analysis Two-Factor Designs CS 147: Computer Systems Performance Analysis Two-Factor Designs 1 / 34

Overview CS147 Overview 2015-06-15 Two-Factor Designs No Replications Adding Replications Overview Two-Factor Designs No Replications Adding Replications 2 / 34

Two-Factor Designs No Replications Two-Factor Design Without Replications CS147 Two-Factor Design Without Replications 2015-06-15 Two-Factor Designs ◮ Used when only two parameters, but multiple levels for each No Replications ◮ Test all combinations of levels of the two parameters ◮ One replication (observation) per combination ◮ For factors A and B with a and b levels, ab experiments required Two-Factor Design Without Replications ◮ Used when only two parameters, but multiple levels for each ◮ Test all combinations of levels of the two parameters ◮ One replication (observation) per combination ◮ For factors A and B with a and b levels, ab experiments required 3 / 34

Two-Factor Designs No Replications When to Use This Design? CS147 When to Use This Design? 2015-06-15 Two-Factor Designs ◮ System has two important factors ◮ Factors are categorical No Replications ◮ More than two levels for at least one factor ◮ Examples: ◮ Performance of different processors under different workloads ◮ Characteristics of different compilers for different benchmarks When to Use This Design? ◮ Performance of different Web browsers on different sites ◮ System has two important factors ◮ Factors are categorical ◮ More than two levels for at least one factor ◮ Examples: ◮ Performance of different processors under different workloads ◮ Characteristics of different compilers for different benchmarks ◮ Performance of different Web browsers on different sites 4 / 34

Two-Factor Designs No Replications When to Avoid This Design? CS147 When to Avoid This Design? 2015-06-15 Two-Factor Designs ◮ Systems with more than two important factors ◮ Use general factorial design No Replications ◮ Non-categorical variables ◮ Use regression ◮ Only two levels per factor ◮ Use 2 2 designs When to Avoid This Design? ◮ Systems with more than two important factors ◮ Use general factorial design ◮ Non-categorical variables ◮ Use regression ◮ Only two levels per factor ◮ Use 2 2 designs 5 / 34

Two-Factor Designs No Replications Model For This Design CS147 Model For This Design 2015-06-15 Two-Factor Designs ◮ y ij = µ + α j + β i + e ij ◮ y ij is observation No Replications ◮ µ is mean response ◮ α j is effect of factor A at level j ◮ β i is effect of factor B at level i Model For This Design ◮ e ij is error term ◮ Sums of α j ’s and β i ’s are both zero ◮ y ij = µ + α j + β i + e ij ◮ y ij is observation ◮ µ is mean response ◮ α j is effect of factor A at level j ◮ β i is effect of factor B at level i ◮ e ij is error term ◮ Sums of α j ’s and β i ’s are both zero 6 / 34

Two-Factor Designs No Replications Assumptions of the Model CS147 Assumptions of the Model 2015-06-15 Two-Factor Designs ◮ Factors are additive ◮ Errors are additive No Replications ◮ Typical assumptions about errors: ◮ Distributed independently of factor levels ◮ Normally distributed Assumptions of the Model ◮ Remember to check these assumptions! ◮ Factors are additive ◮ Errors are additive ◮ Typical assumptions about errors: ◮ Distributed independently of factor levels ◮ Normally distributed ◮ Remember to check these assumptions! 7 / 34

Two-Factor Designs No Replications Computing Effects CS147 Computing Effects 2015-06-15 Two-Factor Designs ◮ Need to figure out µ , α j , and β i No Replications ◮ Arrange observations in two-dimensional matrix ◮ b rows, a columns ◮ Compute effects such that error has zero mean ◮ Sum of error terms across all rows and columns is zero Computing Effects ◮ Need to figure out µ , α j , and β i ◮ Arrange observations in two-dimensional matrix ◮ b rows, a columns ◮ Compute effects such that error has zero mean ◮ Sum of error terms across all rows and columns is zero 8 / 34

Two-Factor Designs No Replications Two-Factor Full Factorial Example CS147 Two-Factor Full Factorial Example 2015-06-15 Two-Factor Designs ◮ Want to expand functionality of a file system to allow automatic compression ◮ Examine three choices: No Replications ◮ Library substitution of file system calls ◮ New VFS ◮ Stackable layers Two-Factor Full Factorial Example ◮ Three different benchmarks ◮ Metric: response time ◮ Want to expand functionality of a file system to allow automatic compression ◮ Examine three choices: ◮ Library substitution of file system calls ◮ New VFS ◮ Stackable layers ◮ Three different benchmarks ◮ Metric: response time 9 / 34

Two-Factor Designs No Replications Data for Example CS147 Data for Example 2015-06-15 Two-Factor Designs Library VFS Layers Compile 94.3 89.5 96.2 No Replications Benchmark Email 224.9 231.8 247.2 Benchmark Web Server 733.5 702.1 797.4 Data for Example Benchmark Library VFS Layers Compile 94.3 89.5 96.2 Benchmark Email 224.9 231.8 247.2 Benchmark Web Server 733.5 702.1 797.4 Benchmark 10 / 34

Two-Factor Designs No Replications Computing µ CS147 Computing µ 2015-06-15 Two-Factor Designs ◮ Averaging the j th column, y · j = µ + α j + 1 β i + 1 � � e ij No Replications b b i i ◮ By assumption, error terms add to zero ◮ Also, the β j ’s add to zero, so y · j = µ + α j Computing µ ◮ Averaging rows produces y i · = µ + β i ◮ Averaging everything produces y ·· = µ ◮ Averaging the j th column, y · j = µ + α j + 1 β i + 1 � � e ij b b i i ◮ By assumption, error terms add to zero ◮ Also, the β j ’s add to zero, so y · j = µ + α j ◮ Averaging rows produces y i · = µ + β i ◮ Averaging everything produces y ·· = µ 11 / 34

Two-Factor Designs No Replications Model Parameters CS147 Model Parameters 2015-06-15 Two-Factor Designs Using same techniques as for one-factor designs, parameters are: No Replications ◮ y ·· = µ ◮ α j = y · j − y ·· ◮ β i = y i · − y ·· Model Parameters Using same techniques as for one-factor designs, parameters are: ◮ y ·· = µ ◮ α j = y · j − y ·· ◮ β i = y i · − y ·· 12 / 34

Two-Factor Designs No Replications Calculating Parameters for the Example CS147 Calculating Parameters for the Example 2015-06-15 Two-Factor Designs ◮ µ = grand mean = 357.4 ◮ α j = ( − 6 . 5 , − 16 . 3 , 22 . 8 ) No Replications ◮ β i = ( − 264 . 1 , − 122 . 8 , 386 . 9 ) ◮ So, for example, the model predicts that the email benchmark using a special-purpose VFS will take Calculating Parameters for the Example 357 . 4 − 16 . 3 − 122 . 8 = 218 . 3 seconds ◮ µ = grand mean = 357.4 ◮ α j = ( − 6 . 5 , − 16 . 3 , 22 . 8 ) ◮ β i = ( − 264 . 1 , − 122 . 8 , 386 . 9 ) ◮ So, for example, the model predicts that the email benchmark using a special-purpose VFS will take 357 . 4 − 16 . 3 − 122 . 8 = 218 . 3 seconds 13 / 34

Two-Factor Designs No Replications Estimating Experimental Errors CS147 Estimating Experimental Errors 2015-06-15 Two-Factor Designs ◮ Similar to estimation of errors in previous designs No Replications ◮ Take difference between model’s predictions and observations ◮ Calculate Sum of Squared Errors ◮ Then allocate variation Estimating Experimental Errors ◮ Similar to estimation of errors in previous designs ◮ Take difference between model’s predictions and observations ◮ Calculate Sum of Squared Errors ◮ Then allocate variation 14 / 34

Two-Factor Designs No Replications Allocating Variation CS147 Allocating Variation 2015-06-15 Two-Factor Designs ◮ Use same kind of procedure as on other models No Replications ◮ SSY = SS0 + SSA + SSB + SSE ◮ SST = SSY − SS0 ◮ Can then divide total variation between SSA, SSB, and SSE Allocating Variation ◮ Use same kind of procedure as on other models ◮ SSY = SS0 + SSA + SSB + SSE ◮ SST = SSY − SS0 ◮ Can then divide total variation between SSA, SSB, and SSE 15 / 34

Two-Factor Designs No Replications Calculating SS0, SSA, SSB CS147 Calculating SS0, SSA, SSB 2015-06-15 Two-Factor Designs ◮ SS0 = ab µ 2 No Replications ◮ SSA = b � j α 2 j ◮ SSB = a � i β 2 i ◮ Recall that a and b are numbers of levels for the factors Calculating SS0, SSA, SSB ◮ SS0 = ab µ 2 j α 2 ◮ SSA = b � j ◮ SSB = a � i β 2 i ◮ Recall that a and b are numbers of levels for the factors 16 / 34

Two-Factor Designs No Replications Allocation of Variation for Example CS147 Allocation of Variation for Example 2015-06-15 Two-Factor Designs ◮ SSE = 2512 ◮ SSY = 1 , 858 , 390 ◮ SS0 = 1 , 149 , 827 No Replications ◮ SSA = 2489 ◮ SSB = 703 , 561 ◮ SST = 708 , 562 Allocation of Variation for Example ◮ Percent variation due to A: 0.35% ◮ Percent variation due to B: 99.3% ◮ Percent variation due to errors: 0.35% ◮ SSE = 2512 ◮ SSY = 1 , 858 , 390 ◮ SS0 = 1 , 149 , 827 ◮ SSA = 2489 ◮ SSB = 703 , 561 ◮ SST = 708 , 562 ◮ Percent variation due to A: 0.35% ◮ Percent variation due to B: 99.3% ◮ Percent variation due to errors: 0.35% 17 / 34