parameter tuning for search based test data generation
play

Parameter Tuning for Search-Based Test-Data Generation Revisited - PowerPoint PPT Presentation

Aug 10, 2023 •146 likes •725 views

Parameter Tuning for Search-Based Test-Data Generation Revisited Support for Previous Results Anton Kotelyanskii Gregory M. Kapfhammer creative commons licensed ( BY-NC-ND ) ickr photo shared by sunface13 Software Testing Software

  1. Parameter Tuning for Search-Based Test-Data Generation Revisited Support for Previous Results Anton Kotelyanskii Gregory M. Kapfhammer creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  2. Software Testing

  3. Software Testing Test Suites

  4. Software Testing Test Suites Automatic Generation

  5. Software Testing Test Suites Automatic Generation Confronting Challenges

  6. Software Testing Test Suites Automatic Generation Confronting Challenges Evaluation Strategies

  7. Empirical Studies

  8. Empirical Studies Challenges

  9. Empirical Studies Challenges Importance

  10. Empirical Studies Challenges Importance Replication

  11. Empirical Studies Challenges Importance Replication Rarity

  13. EvoSuite creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  14. EvoSuite Amazing test suite generator creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  15. EvoSuite Amazing test suite generator Uses a genetic algorithm creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  16. EvoSuite Amazing test suite generator Uses a genetic algorithm Input : A Java class creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  17. EvoSuite Amazing test suite generator Uses a genetic algorithm Input : A Java class Output : A JUnit test suite creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  18. EvoSuite Amazing test suite generator Uses a genetic algorithm Input : A Java class Output : A JUnit test suite http://www.evosuite.org/ creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  19. Parameter Tuning

  20. Parameter Tuning RSM : Response surface methodology

  21. Parameter Tuning RSM : Response surface methodology SPOT : Sequential parameter optimization toolbox

  22. Parameter Tuning RSM : Response surface methodology SPOT : Sequential parameter optimization toolbox Successfully applied to many diverse problems!

  23. Defaults or Tuned Values?

  25. Experiment Design creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  26. Experiment Design Eight EvoSuite parameters creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  27. Experiment Design Eight EvoSuite parameters Ten projects from SF100 creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  28. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  29. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  30. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning Aiming to improve statement coverage creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  31. Parameters Parameter Name Minimum Maximum Population Size 5 99 Chromosome Length 5 99 Rank Bias 1.01 1.99 Number of Mutations 1 10 Max Initial Test Count 1 10 Crossover Rate 0.01 0.99 Constant Pool Use Probability 0.01 0.99 Test Insertion Probability 0.01 0.99

  32. Experiments

  33. Experiments 184 days of computation time estimated

  34. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks

  35. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi�ed 139 "easy" and 21 "hard" classes

  36. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi�ed 139 "easy" and 21 "hard" classes Mann-Whitney U-test and

  37. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi�ed 139 "easy" and 21 "hard" classes Mann-Whitney U-test and Vargha-Delaney e�ect size

  38. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

  39. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage

  40. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage E�ect size greater than 0.5 means that tuning is worse

  41. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage E�ect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis

  42. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage E�ect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis Additional empirical results in the QSIC 2014 paper!

  44. Discussion creative commons licensed ( BY ) photo shared by Startup Stock Photos

  45. Discussion Tuning improved scores for 11 classes creative commons licensed ( BY ) photo shared by Startup Stock Photos

  46. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults creative commons licensed ( BY ) photo shared by Startup Stock Photos

  47. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft �oor" may exist for parameter tuning creative commons licensed ( BY ) photo shared by Startup Stock Photos

  48. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft �oor" may exist for parameter tuning Additional details in the QSIC 2014 paper! creative commons licensed ( BY ) photo shared by Startup Stock Photos

  49. Practical Implications

  50. Practical Implications Fundamental Challenges

  51. Practical Implications Fundamental Challenges Tremendous Con�dence

  52. Practical Implications Fundamental Challenges Tremendous Con�dence Great Opportunities

  54. Important Contributions creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  55. Important Contributions Comprehensive Experiments creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  56. Important Contributions Comprehensive Experiments Conclusive Con�rmation creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  57. Important Contributions Comprehensive Experiments Conclusive Con�rmation For EvoSuite, Defaults = Tuned creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

Recommend

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning? What we have seen so far many learning algorithms for classification, regression, ... Many of those have parameters k and distance

797 views • 52 slides
Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning? What we have seen so far many learning algorithms for classification, regression, ... Many of those have parameters k and distance

586 views • 54 slides
Search Based Test Data Generation for Server-side Web Application Testing Nadia Alshahwan and

Search Based Test Data Generation for Server-side Web Application Testing Nadia Alshahwan and

Search Based Test Data Generation for Server-side Web Application Testing Nadia Alshahwan and Mark Harman CREST Centre University College London Automated web application testing using search based software Engineering (ASE 2011) Hill

588 views • 15 slides
Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet

Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet

Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet University of Glasgow [s.jouet.1@research.gla.ac.uk] Background TCP CongesHon Control For a transport endpoint embedded in a

546 views • 9 slides
On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg

On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg

On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg and Patrick Johnson Artifjcially Intelligent Manufacturing Center University of Wyoming larsko,pjohns27@uwyo.edu UCC, 02 April 2019 1 Big

567 views • 41 slides
S. Scalabrino G. Grano D. Di Nucci A. De Lucia M. Guerra H. Gall R. Oliveto OCELOT a

S. Scalabrino G. Grano D. Di Nucci A. De Lucia M. Guerra H. Gall R. Oliveto OCELOT a

S. Scalabrino G. Grano D. Di Nucci A. De Lucia M. Guerra H. Gall R. Oliveto OCELOT a Search-Based Test-Data Generation Tool for C Testing is expensive Automatic Test Generation Search-Based Test Generation TIOBE Ranking TIOBE Ranking

398 views • 28 slides
10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy

10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy

10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy Parameter Tuning vs Parameter Control EA calibration Parameter Tuning Testing Effort Parameters and Recommendations Parameter

569 views • 11 slides
Model-Based Testing: Automated Generation of Test Cases, Test Data, and Test Procedures from

Model-Based Testing: Automated Generation of Test Cases, Test Data, and Test Procedures from

Model-Based Testing: Automated Generation of Test Cases, Test Data, and Test Procedures from SysML Models Jrg Brauer and Jan Peleska Verified Systems International GmbH support@verified.de Space Tech Expo Europe 2015-11-19 Motivation

499 views • 26 slides
MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas

MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas

MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas Fellner FMCAD Student Forum Wien, October 4th 2017 TEST CASE GENERATION WITH 1 Andreas Fellner TEST CASE GENERATION WITH Abstract Model UML /

358 views • 34 slides
A UML-Based Testing Approach Using Sequence Diagrams, Statecharts, and OCL Constraints Dehla

A UML-Based Testing Approach Using Sequence Diagrams, Statecharts, and OCL Constraints Dehla

A UML-Based Testing Approach Using Sequence Diagrams, Statecharts, and OCL Constraints Dehla Sokenou TU Berlin Softwaretechnik Overview of the Test System test data test generation driver test case abstract test data generation test

444 views • 20 slides
Pruning the Search Space in Path-based Test Generation Motivation SE S ebastien Bardin

Pruning the Search Space in Path-based Test Generation Motivation SE S ebastien Bardin

Pruning the Search Space in Path-based Test Generation Motivation SE S ebastien Bardin Heuristics sebastien.bardin@cea.fr Experiments CEA-LIST, Software Security Labs Conclusion (joint work with Philippe Herrmann) S ebastien Bardin,

837 views • 44 slides
Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and

Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and

Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and time-consuming Algorithm configuration using Machine Learning Focus on tree search algorithms Branch-and-Bound Tree Search Widely

396 views • 17 slides
Automated Test Data Generation for Coverage: Lakhotia et al. Outline Havent We Solved This

Automated Test Data Generation for Coverage: Lakhotia et al. Outline Havent We Solved This

Havent We Solved This Problem Yet? Automated Test Data Generation for Coverage: Lakhotia et al. Outline Havent We Solved This Problem Yet? Motivation Background Search Based Kiran Lakhotia 1 Phil McMinn 2 Mark Harman 1 Testing

811 views • 46 slides
Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product

Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product

Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product Development Team Joint MSC/Stakeholder Meeting September 25, 2008 Uneconomic Adjustment Policy/Parameter Tuning Overview/Context Greg Cook Manager,

876 views • 66 slides
Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R Voter dataset from US 2016 election Split intro training and test set

779 views • 22 slides
Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance

Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance

Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance Committee (MSC) Joint Stakeholder/MSC Meeting July 30, 2008 Common Features of Scarcity Pricing and Parameter Tuning Scarcity pricing occurs

731 views • 13 slides
Towards Constraint Logic Programming over Strings for Test Data Generation Sebastian Krings, J.

Towards Constraint Logic Programming over Strings for Test Data Generation Sebastian Krings, J.

Towards Constraint Logic Programming over Strings for Test Data Generation Sebastian Krings, J. Schmidt, P. Skowronek, J. Dunkelau, D. Ehmke Test Data vs. Privacy Software testing needs Personal data should has appropriate test data to be

212 views • 18 slides
Scalable Bandit Methods for Hyper-parameter Tuning Kirthevasan Kandasamy Carnegie Mellon

Scalable Bandit Methods for Hyper-parameter Tuning Kirthevasan Kandasamy Carnegie Mellon

Scalable Bandit Methods for Hyper-parameter Tuning Kirthevasan Kandasamy Carnegie Mellon University Guest Lecture - Scalable Machine Learning for Big Data Biology University of Pittsburgh, Pittsburgh, PA November 3, 2017 Hyper-parameter

1.42k views • 127 slides
Parameter Tuning. Automatic Algorithm Configuration Petr Po s k P. Po s k c

Parameter Tuning. Automatic Algorithm Configuration Petr Po s k P. Po s k c

CZECH TECHNICAL UNIVERSITY IN PRAGUE Faculty of Electrical Engineering Department of Cybernetics Parameter Tuning. Automatic Algorithm Configuration Petr Po s k P. Po s k c 2016 A0M33EOA: Evolutionary Optimization

747 views • 32 slides
Search and Machine Learning Kalyan Veeramachaneni, Jason Ansel, Shoaib Kamil, Jeffrey Bosboom,

Search and Machine Learning Kalyan Veeramachaneni, Jason Ansel, Shoaib Kamil, Jeffrey Bosboom,

Search and Machine Learning Kalyan Veeramachaneni, Jason Ansel, Shoaib Kamil, Jeffrey Bosboom, Una-May OReilly, Saman Amarasinghe CGO Tutorial February 8th, 2015 Overview Parameter types and tuning An example tuning problem -

371 views • 35 slides
On Information-Maximization On Information-Maximization Clustering: Tuning Parameter Clustering:

On Information-Maximization On Information-Maximization Clustering: Tuning Parameter Clustering:

ICML2011 Jun. 28-Jul. 2, 2011 On Information-Maximization On Information-Maximization Clustering: Tuning Parameter Clustering: Tuning Parameter Selection and Analytic Solution Selection and Analytic Solution Masashi Sugiyama, Makoto Yamada,

773 views • 28 slides
Parameter Tuning for Influence Maximization Manqing Ma Last Updated: 11/19/2018 (CSCI6250 FNS

Parameter Tuning for Influence Maximization Manqing Ma Last Updated: 11/19/2018 (CSCI6250 FNS

Parameter Tuning for Influence Maximization Manqing Ma Last Updated: 11/19/2018 (CSCI6250 FNS Presentation) Outline Objective: Param Tuning for BI /GPI ( ref: Karampourniotis, P. D., Szymanski, B. K., & Korniss, G. (2018). Influence

535 views • 20 slides
Lecture 3: Method evaluation and tuning parameter selection Felix Held, Mathematical Sciences

Lecture 3: Method evaluation and tuning parameter selection Felix Held, Mathematical Sciences

Lecture 3: Method evaluation and tuning parameter selection Felix Held, Mathematical Sciences MSA220/MVE440 Statistical Learning for Big Data 29th March 2019 Evaluating performance of a statistical method Goals structure, e.g. in kNN

716 views • 27 slides
A Short Introduction to Bayesian Optimization With applications to parameter tuning on

A Short Introduction to Bayesian Optimization With applications to parameter tuning on

A Short Introduction to Bayesian Optimization With applications to parameter tuning on accelerators Johannes Kirschner 28th February 2018 ICFA Workshop on Machine Learning for Accelerator Control Solve x = arg max f ( x ) x X 0

511 views • 36 slides

More recommend