Motivation Motivation 2 Typical e-commerce sites are complex, - PDF document

A METHOD FOR A METHOD FOR EVALUATING THE IMPACT OF SOFTWARE CONFIGURATION PARAMETERS ON E PARAMETERS ON E- COMMERCE SITES Monchai Sopitkamol Daniel A. Menascé George Mason University Kasetsart University Dept. of Computer Science, MS 4A5 Dept. of Computer Engineering 4400 University Drive 4400 University Drive B Bangkok, Thailand k k Th il d Fairfax, Virginia fengmcs@ku.ac.th menasce@cs.gmu.edu WOSP 2005, Spain Motivation Motivation 2 � Typical e-commerce sites are complex, consisting of hundreds of machines with a large number of configurable parameters that may take many different values � The number of possible combinations of the values of these parameters can be extremely large � An e-commerce site’s performance can be significantly impacted by the settings of these significantly impacted by the settings of these parameters

The Problem The Problem 3 � There is a need to study the performance impact of the myriad of configurable parameters and their values for multi-tiered e-commerce sites. � The knowledge of which factors and their levels are g relevant is critical and can be used to guide performance tuning efforts as well as the design of performance tuning efforts as well as the design of self-optimizing e-commerce sites. Contributions of the Paper Contributions of the Paper 4 � Design of a practical, ad-hoc experimental methodology D i f ti l d h i t l th d l � For evaluating the performance impact of parameters, their interactions, and for ranking the parameters � Design of Experiments � Conducted experiments on a site compliant with the TPC-W benchmark for e-commerce � Experimental Data Analysis � Use of ANOVA models to test various single-parameter and parameter interaction hypotheses interaction hypotheses � Factor Rankings � Designed a ranking methodology � Ranked factors for individual e-commerce functions � Ranked factors for all e-commerce functions combined

Outline Outline 5 � Design of the Experiments � Workload and Parameters � Research Methodology � Experimental Results � E i t l R lt � Performance Engineering Guidelines Design of Experiments Design of Experiments 6 � Response Variables R V i bl � Web Interaction Response Time (WIRT) � Web Interaction per Second (WIPS) � Web Interaction per Second (WIPS) � Probability of Rejection (P rej ) � Factors � Factors � Factor A (Web Interaction Types): consists of 14 levels and is prescribed by TPC-W specification � Factor B (Configurable factors): consists of 28 factors and f f f f stems from the three-tiered architecture � Levels � Levels � Finite: discrete factors � Infinite: continuous and range factors g

Workload Characterization… Factor A Workload Characterization… Factor A 7 Web Interaction Type Frequency Search Request 20.00% Search Results 17.00% Product Detail Product Detail 17.00% 17 00% Home 16.00% Shopping Cart 11.60% New Products 5.00% Best Sellers 5.00% Cus o Customer Registration e eg s a o 3.00% 3 00% Buy Request * 2.60% Buy Confirm * 1.20% Order Inquiry * * O 0.75% % Order Display * 0.66% Admin Request q 0.10% Admin Confirm 0.09% * = An SSL-connection Web interaction Configurable Factors Configurable Factors… Factor B F t B Application Server Application Server Database Server Database Server Web Server (IIS 5.0) (Tomcat 4.1) (SQL Server 7.0) HTTP KeepAlive acceptCount Cursor Threshold Application Protection Level A li i P i L l minProcessors i P Fill F Fill Factor Connection Timeout maxProcessors Locks Number of Connections Max Worker Threads Logging Location Min Memory Per Query Resource Indexing Network Packet Size Performance Tuning Level Priority Boost Application Optimization Recovery Interval MemCacheSize Set Working Set Size MaxCachedFileSize Max Server Memory ListenBacklog Min Server Memory MaxPoolThreads User Connections worker ajp13 cachesize worker.ajp13.cachesize 8

Experimental Methodology Experimental Methodology 9 Experimental Methodology Experimental Methodology 10

Experimental Methodology Experimental Methodology… Initialization I i i li i 11 Goal : To find a “good” level of each factor so that the Search Request interaction exhibits the smallest possible response times Factors A B C D Levels Levels (a 1 , a 2 ) (a 1 , a 2 ) (b 1 , b 2 ) (b 1 , b 2 ) (c 1 , c 2 ) (c 1 , c 2 ) (d 1 , d 2 ) (d 1 , d 2 ) Pass One Pass Two * * * * a a c c a a b b c c d d 1) 1) (A= , B=?, C= , D=?) (A= B=? C= D=?) 1) (A= , B= , C= , D= ) 1) (A= B= C= D= ) 1 2 1 2 2 2 * * * * a c a c d d 2) ( , B=?, , D= ) 2) ( , B=?, , D= ) 1 2 1 2 2 1 * * * * a a b b c c a a b b c c 3) ( , B= , , D=?) 3) ( B D ?) 3) ( , B= , , D=?) 3) ( B D ?) 1 1 2 2 1 2 * * * * b d a c a b c d 4) ( , B= , , D= ) 4) ( , B= , , D= ) 1 2 2 2 1 1 2 2 Initialization Example Experimental Methodology Experimental Methodology… Initialization I i i li i 12 Initialization Algorithm for SUT

Experimental Methodology Experimental Methodology 13 Experimental Methodology… Data Collection 14 Goal : To generate experimental data for all factors for subsequent statistical analyses (hypothesis tests and factor rankings) Factors A B C D Levels Levels (a 1 , a 2 ) (a 1 , a 2 ) (b 1 , b 2 ) (c 1 , c 2 ) (d 1 , d 2 ) (b 1 , b 2 ) (c 1 , c 2 ) (d 1 , d 2 ) * * d a b c Initialization Results (A= , B= , C= , D= ) 2 2 1 1 Factor A ⇒ (*.*, b 1 , c 2 *, d 2 ) ( , 1 , 2 , 2 ) Factor B ⇒ (a 1 *, *.*, c 2 *, d 2 ) Factor C ⇒ (a 1 *, b 1 , *.*, d 2 ) Factor D ⇒ (a 1 *, b 1 , c 2 , *.*) Data Collection Example

Experimental Methodology… Data Collection Experimental Methodology… Data Collection Algorithm for the Data Collection Process 15 Experimental Methodology Experimental Methodology 16

Experimental Methodology… Hypothesis 1 17 Hypothesis 1 : “No difference due to each configurable factor B” μ = μ = = μ � H : , = number of levels of a factor c 0 . 1 . . 2 . . c . Example: Factor B = HTTP Keep-Alive H 0 : μ .1. = μ .2 . F-Test statistic for Factor B: F = MSB/MSE Reject H 0 if F > Fu (the upper-tail critical value from an F distribution with ( c -1 = 1) 0 degrees of freedom in numerator and ( n-cr = n-28 ) degrees of freedom in denominator) That is different levels (enabled and disabled) of HTTP Keep Alive options yield That is, different levels (enabled and disabled) of HTTP Keep-Alive options yield different system performance back Experimental Methodology… Hypothesis 2 18 Hypothesis 2: “No interaction effects between each configurable factor (B) and fourteen Web interaction types (A)” (B) and fourteen Web interaction types (A) H 0 : The interaction of A and B = 0 Example: Factor B = HTTP Keep-Alive H 0 : The interaction of A and B = 0 F-Test statistic for the interaction: F = MSAB/MSE Reject H 0 if F > Fu (the upper-tail critical value from an F distribution with ( r -1)( c -1) 0 = (13)(1) = 13 degrees of freedom in numerator and ( n-cr = n-28 ) degrees of freedom in denominator) That is there is interaction between HTTP Keep Alive factor and the fourteen Web That is, there is interaction between HTTP Keep-Alive factor and the fourteen Web Interaction types. back

Experimental Methodology… Hypothesis 3 19 Hypothesis 3 : “No difference due to each configurable factor B for each of the five most important Web interaction types” of the five most important Web interaction types μ = μ = = μ � H 0 : 1 2 c Example: Factor B = HTTP Keep-Alive H 0 : μ . 1. = μ .2 . F-Test statistic for Factor B: F = MSB/MSE Reject H 0 if F > Fu (the upper-tail critical value from an F distribution with ( c -1 = 1) 0 degrees of freedom in numerator and ( n-cr = n-28 ) degrees of freedom in denominator) That is different levels (enabled and disabled) of HTTP Keep Alive options yield That is, different levels (enabled and disabled) of HTTP Keep-Alive options yield different system performance for certain Web interaction type back Experimental Methodology… p gy Applying the ANOVA models � Three null hypothesis tests � Hypothesis 1 : “No difference due to each configurable factor B” each configurable factor B � Hypothesis 2 : “No interaction effects between each configurable factor (B) b t h fi bl f t (B) and 14 Web interaction types (A)” � Hypothesis 3 : “No difference due to each factor for each of the five most important Web interaction types” p yp 20