Can a Chatbot Support Software Engineers with Load Testing? - PowerPoint PPT Presentation

Can a Chatbot Support Software Engineers with Load Testing? Approach and Experiences Presentation at ACM/SPEC ICPE 2020, Apr. 2020 Samuel Beck @samu_beck Lasse Merz Christoph Zorn Fabian Beck @beck_fabian Leonel Merino @leonel_merino Dušan Okanović @okanovic_d André van Hoorn @andrevanhoorn

Motivation Performance Evaluation System ChatBot Eve Colin Mia 2 Can a Chatbot Support Software Engineers with Load Testing?

Goals ● Evaluate the applicability of a chatbot to facilitate performance evaluations. ● Gather impressions from performance experts and novices about using a chatbot in this context. I want to perform a Perform a load test with I want to test Eve’s load test for Eve’s 100 users and measure system. system. response time. 3 Can a Chatbot Support Software Engineers with Load Testing?

Approach User Interaction Language Processing Analysis & Reporting Eve Mia Colin PerformoBot 4 Can a Chatbot Support Software Engineers with Load Testing?

Load test intents and entities Intent: Load test Newbie Output context: loadtest Entities: ● Analysis type Training Phrases: Please perform a load test ● I want to create a load test. for me. ● Help me to define a load test. Great, let us set up a load test together. I ● I want to perform a load test. need to know what load you want to use, the domain to be tested and the metric which should be measured. Intent: Define a load test Output context: loadtest Record the response time for 100 users on Entities: www.example.com. ● Load ● Domain ● Metric Alright I created a load test for you with ● Analysis type (optional) the following parameters: load: 100 users, ● Duration (optional) domain: www.example.com, metric: response time. If you are satisfied I can Training Phrases: ● execute the load test. Otherwise, you can Loadtest with 55 users ● I want one with 3 hours, 22 users change the parameters, add more and response time. parameters from the parameter list or ● Create a load test with 30 users at reset and start over. www.myservice.de and measure server status. 5 Can a Chatbot Support Software Engineers with Load Testing?

Demo 6 Can a Chatbot Support Software Engineers with Load Testing?

Demo 7 Can a Chatbot Support Software Engineers with Load Testing?

Performance Report 1 3 2 8 4 Can a Chatbot Support Software Engineers with Load Testing?

Study : Research Questions How can a chatbot help users do a 01 Task performance evaluation through a load test? + How do users interact with 02 PerformoBot? Survey What educational effect does 03 PerformoBot have on participants? What are the differences between 04 inexperienced users and experts in 47 participants the above categories? 9 Can a Chatbot Support Software Engineers with Load Testing?

Study: Task + Survey Test System Participant evaluate 8:00 min PerformoBot Survey 10 Can a Chatbot Support Software Engineers with Load Testing?

Study: Results Median of 10 Average of interactions per 47 participants 3.64 minutes participant session time No educational 87.2% finished effect the task Too little guidance Automatic execution 80.9% correctly fallback mechanisms & answered the & report creation scenario missing keywords Complex load testing capabilities Explanations 11 Can a Chatbot Support Software Engineers with Load Testing?

Study: Inexperienced Users vs. Experts Helpfulness of Sentiment PerformoBot 22 novices and Very much positive neutral negative 25 experts 18.2% 48.0% Novices and 18.2% experts were equally able to 8.0% successfully complete the task 63.7% 44.0% Novices evaluated PerformoBot more positive and Not very much helpful 12 Can a Chatbot Support Software Engineers with Load Testing?

Conclusion Mia Colin ● More performance concerns Helpful for novice/inexperienced users ● Improved usability features Artifacts: Eve Needs advanced functionality for experts 13 Can a Chatbot Support Software Engineers with Load Testing?

References Icons from flaticon (https://www.flaticon.com/) made by dave-gandy (question mark), monkik (people), freepik, eucalyp, smashicons (analysis, computers, clocks), flat-icons (performobot), roundicons (check mark, x, smileys), and from https://icons8.de. Chatbot gif: https://cdn.dribbble.com/users/37530/screenshots/2937858/drib_blink_bot.gif Messages gif: https://i.gifer.com/origin/98/98447b873b927d46f752e9e0fc9c2910_w200.gif 14 Can a Chatbot Support Software Engineers with Load Testing?