Web Application Stress Testing with Blaise Internet with Blaise Internet Jim O’Reilly Jim O Reilly Westat International Blaise Users Conference Annapolis Maryland, USA September 28, 2007 Septe be 8, 00
Introduction � F Focus: web application stress testing tools, methods, tests and b li ti t t ti t l th d t t d lessons learned � Benefits of stress testing � Realistic simulation of performance of server and application under varying numbers of concurrent users � Valuable information on suitability of the server configuration for specific requirements of a survey � Potential in usability testing and paradata analysis � Test results presented here highly dependent on p g y p � Size and complexity of the data models � Server platform � Your mileage may vary � Your mileage may vary 04/ 10/ 2008 2
Planning and Configuring a Web Survey � Calculating server capacity essential C l l ti it ti l � Respondent cooperation crucial � Web surveys often long and complex � Web surveys often long and complex � Lags processing pages on server drive Rs away � Server capacity complex Server capacity complex � Processors, memory, speed etc. � Application performance that meets needs of study is core � Done by stress testing � Blaise 4.8 internet system enhancements offer improved scalability -- widening potential uses of Blaise l bilit id i t ti l f Bl i 04/ 10/ 2008 3
Peak Concurrent Users � � Key metric for server configuration Key metric for server configuration Klaus Salchner : http://aspnet.4guysfromrolla.com/articles/062304-1.aspx#postadlink � Peak Peak = three times average concurrent users three times a erage conc rrent sers 04/ 10/ 2008 4
Microsoft Web Application Stress Tool � Freeware, unsupported by Microsoft F t d b Mi ft � Google “web application stress testing” for links � Capable adaptable suits Blaise testing situations � Capable, adaptable, suits Blaise testing situations 04/ 10/ 2008 5
How WAS works R Runs on user workstation using MS Access database k t ti i MS A d t b � Records web application session � Captures all http traffic Captures all http traffic � Messages generated by user and responses returned � Testing a script � Set parameters--# of users; test times, etc. � As script runs, WAS records timing and result codes of script-server interactions p � Reports on tests generated � One workstation can simulate hundreds of web users web users 04/ 10/ 2008 6
Stress Test Methodology � Follow Blaise team approach reported in Blaise 4.8 F ll Bl i t h t d i Bl i 4 8 Online Assistant at “Test results for web server performance” p � Key measure: increase in waiting time per page between � Baseline WAS test with one user and � Tests of incrementing levels of concurrent users 04/ 10/ 2008 7
Test Steps � Record Blaise Internet survey session script with WAS R d Bl i I t t i i t ith WAS � In WAS � Set script’s page delay to a constant value reflecting Set script s page delay to a constant value reflecting expected average time users take reading, thinking, entering, and pressing Next � 10 seconds per field on a page used (for pages that � 10 seconds per field on a page used (for pages that post to the IS page handler) � Run � benchmark single user test for one hour � successive one hour tests increasing # of users � Extract count of pages sent from reports Extract count of pages sent from reports � Calculate results for session series 04/ 10/ 2008 8
Test series Table 1: Data models & WAS scripts Table 1: Data models & WAS scripts Si Signals/ l / BMI i BMI size IS IS S Script i t Fields Checks (kb) Pages Pages BRFS BRFS 41 41 0 0 18 18 20 20 17 17 Wes1 640 29 543 83 31 Wes2 370 22 297 221 177 Server: one Intel Xeon 1.3 Ghz processor and 1Gb of RAM, running Windows 2003 Server 04/ 10/ 2008 9
Table 2: Stress test of BRFS (60 minutes with average posted page delay of 22.2 seconds) Extra Extra TPP* TPP Users Pages 1 536 6.72 0.0 25 25 13268 13268 6 78 6.78 0 07 0.07 50 25016 7.20 0.48 60 28140 7.68 0.96 70 30580 8.24 1.52 80 30568 9.42 2.71 90 90 30876 30876 10.49 10.49 3.78 3.78 100 25397 14.17 7.46 200 25555 28.17 21.46 250 250 27620 27620 32 59 32.59 25 87 25.87 *TPP = (Min*60*Users)/Pages 04/ 10/ 2008 10
Table 3. Stress Test of Wes1 and Wes2 (60 minutes, average posted page delay of 23.2 & 11.8 seconds) Extra Extra Users Pages TPP* TPP Users Pages TPP* TPP 1 454 7.93 1 373 9.65 5 5 2256 2256 7.98 7.98 0.0 0.0 5 5 1867 1867 9.64 9.64 0.0 0.0 10 4449 8.09 0.2 10 3630 9.92 0.3 15 5561 9.71 0.1 15 6368 8.48 0.6 20 7366 9.77 0.1 20 7659 9.40 1.5 25 8503 10.58 0.9 25 8513 10.57 2.6 30 9296 11.62 2.0 30 8845 12.21 4.3 35 10098 12.48 2.8 35 8882 14.19 6.3 40 10829 13.30 3.6 40 8922 16.14 8.2 45 11454 14.14 4.5 45 9184 17.64 9.7 50 13693 13.15 3.5 50 9336 19.28 11.4 *TPP = (Min*60*Users)/Pages 04/ 10/ 2008 11
Wes1 & Wes2 Test Comparison Increased Seconds Per Page by Number of Users Wes1 and Wes2 Applications 7.0 6.0 5.0 ed TPP 4.0 Wes1 Increase 3.0 3 0 W Wes2 2 2.0 1.0 0.0 5 10 15 20 25 30 35 -1.0 Users/Threads 04/ 10/ 2008 12
W Wes1 & Wes2 Test Comparison (2) 1 & W 2 T t C i (2) � B tt Better Wes2 performance W 2 f � 40% fewer fields, less complex rules, more pages � Follow up plans � Add second VM server running as IS rules server � More powerful server configurations Signals/ BMI size IS Script Fields Checks (kb) ( ) Pages g Pages g BRFS 41 0 18 20 17 Wes1 640 29 543 83 31 Wes2 370 22 297 221 177 04/ 10/ 2008 13
Blaise Team’s Tests � Reported in OLA R t d i OLA � Datamodel of 1900 fields with routing and checks � � Peaked at Peaked at � 50 users with no dedicated rules server � 100 users with one rules server 100 users with one rules server � 150 users with two rules server � Server with two 2.8 GHz processors, 4 GB RAM 04/ 10/ 2008 14
WAS and Other Types of Application Testing � Question raised on how an application would feel to a Q ti i d h li ti ld f l t user if run at the same time as a WAS test. � Ran Wes2 with 15 users and launched another survey Ran Wes2 with 15 users and launched another survey interview interactively. Impressions: � Performance OK -- page delay ~1 second, occasionally 3-4 seconds � Found functional anomaly; in enumerated items, up/down arrow key in browser didn’t work up/down arrow key in browser didn t work � SN unable to replicate the problem � Suggest possible use in usability testing? gg p y g � Check app look and feel under stress 04/ 10/ 2008 15
WAS and COM � WAS h WAS has COM interface; open to automation COM i t f t t ti � Built simple VB app to run series of tests � Enables high volume testing 04/ 10/ 2008 16
WAS Database � D t il d d t Detailed data on tests t t � MS Access 95 � Difficult to work with directly � WAS provides access to script pages to change ResponseDelay etc. � VB/other apps can access directly � Mdb also flexible for reading or export data using SQL Server and p g Crystal Reports etc. 04/ 10/ 2008 17
Other Metrics for Blaise Internet Stress Testing � Bl i Blaise team’s stress testing approach t ’ t t ti h � WAS scripts, fixed page delays, one hour tests, examining time per page at different user levels � Elegant, easy to implement and track, especially with WAS automation � Tracking interviewing information during WAS series � Other possible insights on process � # of completed cases � Stats on interview times – avg min max stddev Stats on interview times avg, min, max, stddev � Journal data on for every page sent from server � Page-level stats of potential value – timing distribution, outliers etc etc – as indicators of page structure issues as indicators of page structure issues 04/ 10/ 2008 18
Linking interviewing and WAS testing � In workshop enable journaling process I k h bl j li � Set data model primary key to tie interview to journal � Modify interview starter page to assign web server Modify interview starter page to assign web server session id as primary key Function CreateOptions(Hosts) ... If UseRunTimeOptions Then ... '*** setting sessionid as primary key Dim sCaseID sCaseID = Session.SessionID RootNode.setAttribute "KeyValue", sCaseID '*** ... End Function 04/ 10/ 2008 19
Conclusions � Stress testing Blaise Internet with WAS system St t ti Bl i I t t ith WAS t � Relatively easy to implement and use � Valuable approach to understanding the expected Valuable approach to understanding the expected performance of Blaise Internet for both specific applications and different server configurations. � Able to demonstrate to clients whether the Blaise web � Able to demonstrate to clients whether the Blaise web survey system is ready to meet their needs. � Potential for � usability testing � processing or paradata analysis with journal information j . 04/ 10/ 2008 20
Recommend
More recommend
