alexander p
play

Alexander P . Conrad , Robert S. Roos , and Gregory M. Kapfhammer - PowerPoint PPT Presentation

Introduction Regression Testing Empirical Study Conclusion Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization Alexander P . Conrad , Robert S. Roos , and Gregory M. Kapfhammer


  1. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite Prioritization CE = 0.5789 3, 1, 2 5 Test Orderings Covered Test Requirements ( C ( T , l )) 4 1,2,3 1,2,3 1,2,3 1,2,3 3 1,3,2 1,3,2 1,3,2 2 2,3,1 2,3,1 1 3,1,2 0 0 5 10 15 Testing Time ( l ) Greedy methods often produce high effectiveness orderings Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 4 / 15

  2. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods T 1 T 3 T 2 T 4 Possible configuration of the coverage report Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  3. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods T 1 T 3 T 2 T 4 R 1 R 2 R 3 Possible configuration of the coverage report Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  4. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods T 1 T 1 T 3 T 3 T 2 T 2 T 4 T 4 R 1 R 1 R 2 R 2 R 3 R 3 Possible configuration of the coverage report Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  5. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 Possible configuration of the coverage report Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  6. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 Execution time of the test cases may mislead greedy Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  7. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 Execution time of the test cases may mislead greedy Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  8. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 2 T 3 T 4 Original ordering has low effectiveness score Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  9. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 2 T 3 T 4 CE ( T ) = 0 . 54 Original ordering has low effectiveness score Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  10. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 2 T 1 T 3 T 2 T 4 T 4 Greedy method constructs suite with marginal improvement Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  11. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 2 T 1 T 3 T 2 T 4 T 4 CE ( T ′ ) = 0 . 55 Greedy method constructs suite with marginal improvement Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  12. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 2 T 1 T 3 T 2 T 4 T 4 CE ( T ′ ) = 0 . 55 Greedy can exhibit high run-times (Jiang et al. ASE 2009) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  13. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 3 T 1 T 1 T 1 T 2 T 2 T 3 T 2 T 4 T 4 T 4 T 3 Genetic algorithm finds a higher quality ordering Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  14. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 1 T 2 T 1 T 2 T 3 T 2 T 4 T 3 T 4 T 4 CE ( T ′ ) = 0 . 63 Genetic algorithm finds a higher quality ordering Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  15. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 1 T 2 T 1 T 2 T 3 T 2 T 4 T 3 T 4 T 4 CE ( T ′ ) = 0 . 63 Genetic algorithm is amenable to parallelization Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  16. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 1 T 2 T 1 T 2 T 3 T 2 T 4 T 3 T 4 T 4 CE ( T ′ ) = 0 . 63 Genetic algorithm supports “human in the loop” Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  17. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods time ( T 1 ) = 1 T 1 T 1 T 1 T 3 T 3 T 3 T 2 T 2 T 2 T 4 T 4 T 4 time ( T 2 ) = 1 time ( T 3 ) = 1 R 1 R 1 R 1 R 2 R 2 R 2 R 3 R 3 R 3 time ( T 4 ) = 2 . 45 T 1 T 3 T 1 T 2 T 1 T 2 T 3 T 2 T 4 T 3 T 4 T 4 CE ( T ′ ) = 0 . 63 Genetic algorithm constructs diverse test orderings Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 5 / 15

  18. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Initialize Fitness Selection Crossover Mutation Randomly create suites by repeatedly shuffling � T 1 , . . . , T n � Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  19. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Initialize Execute Fitness Fitness Selection Selection Crossover Crossover Mutation Mutation Execute the phases until the genetic algorithm stagnates Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  20. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Initialize Execute Execute Fitness Fitness Fitness Selection Selection Selection Crossover Crossover Crossover Mutation Mutation Mutation Operators Use coverage effectiveness in this study - others possible Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  21. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Initialize Execute Execute Fitness Fitness Fitness Selection Selection Selection Crossover Crossover Crossover Mutation Mutation Mutation Operators CE Use coverage effectiveness in this study - others possible Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  22. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Initialize Execute Execute Fitness Fitness Fitness Selection Selection Selection Crossover Crossover Crossover Mutation Mutation Mutation Operators CE APFD APRC Use coverage effectiveness in this study - others possible Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  23. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Selection Selection Selection Selection Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Choose orderings to become parents of next generation Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  24. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Selection Selection Selection Selection Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Operators Choose orderings to become parents of next generation Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  25. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Selection Selection Selection Selection Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Operators ROU TOU TRU Choose orderings to become parents of next generation Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  26. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Fitness Selection Selection Selection Selection Selection Crossover Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Mutation Seven possible operators combine parents to produce children Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  27. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Fitness Selection Selection Selection Selection Selection Crossover Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Mutation Operators Seven possible operators combine parents to produce children Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  28. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Initialize Execute Fitness Fitness Fitness Fitness Fitness Selection Selection Selection Selection Selection Crossover Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Mutation Operators MPX PMX POS Seven possible operators combine parents to produce children Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  29. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Fitness Fitness Selection Selection Selection Selection Selection Selection Crossover Crossover Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Mutation Mutation Six possible operators make random changes to orderings Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  30. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Fitness Fitness Selection Selection Selection Selection Selection Selection Crossover Crossover Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Mutation Mutation Operators Six possible operators make random changes to orderings Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  31. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic Algorithms Execute Execute Execute Execute Execute Initialize Fitness Fitness Fitness Fitness Fitness Fitness Selection Selection Selection Selection Selection Selection Crossover Crossover Crossover Crossover Crossover Crossover Mutation Mutation Mutation Mutation Mutation Mutation Operators ISM IVM SIM Six possible operators make random changes to orderings Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 6 / 15

  32. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Child Density Population Stagnancy Mutation Rate Explored a wide variety of genetic algorithm configurations Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  33. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Child Density Child Density Population Population Stagnancy Stagnancy Mutation Rate Mutation Rate How frequently do we modify individual test orderings ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  34. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Child Density Child Density Population Population Stagnancy Stagnancy Mutation Rate Mutation Rate Parameter Values How frequently do we modify individual test orderings ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  35. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Child Density Child Density Population Population Stagnancy Stagnancy Mutation Rate Mutation Rate Parameter Values 0 . 10 0 . 33 0 . 67 How frequently do we modify individual test orderings ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  36. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Population Population Population Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate How many children should be in the next population ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  37. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Population Population Population Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Parameter Values How many children should be in the next population ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  38. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Population Population Population Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Parameter Values 0 . 50 0 . 75 1 . 00 How many children should be in the next population ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  39. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate How many test suites should exist in the population? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  40. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate Parameter Values How many test suites should exist in the population? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  41. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate Parameter Values 75 150 225 How many test suites should exist in the population? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  42. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Child Density Population Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate Mutation Rate How many generations without fitness improvement ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  43. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Child Density Population Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate Mutation Rate Parameter Values How many generations without fitness improvement ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  44. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Child Density Population Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate Mutation Rate Parameter Values 20 30 40 How many generations without fitness improvement ? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  45. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic Algorithm Possible Configurations Possible Configurations Possible Configurations Possible Configurations Possible Configurations Child Density Child Density Child Density Child Density Child Density Population Population Population Population Population Stagnancy Stagnancy Stagnancy Stagnancy Stagnancy Mutation Rate Mutation Rate Mutation Rate Mutation Rate Mutation Rate Parameter Values 20 30 40 See the paper for further operator and configuration details Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 7 / 15

  46. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 Tree Models: Recursive partitioning creates hierarchical view of data Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  47. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 Tree Models: Recursive partitioning creates hierarchical view of data Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  48. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 Tree Models: Recursive partitioning creates hierarchical view of data Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  49. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 pop_size < 187.5 0.9573 Tree Models: Recursive partitioning creates hierarchical view of data Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  50. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 sel_method child_density < 0.875 0.9674 child_density pop_size pop_size < 187.5 0.9573 Explanatory Variable: Configuration of the genetic algorithm Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  51. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 sel_method sel_method child_density < 0.875 0.9674 child_density pop_size pop_size < 187.5 0.9573 Categorical Non-parametric techniques that handles different variable types Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  52. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 sel_method child_density < 0.875 0.9674 child_density child_density pop_size pop_size pop_size < 187.5 0.9573 Categorical Categorical Numerical Non-parametric techniques that handles different variable types Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  53. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models sel_method: TOU3, TOU4, TOU5 sel_method child_density < 0.875 0.9674 0.9674 child_density pop_size pop_size < 187.5 0.9573 0.9573 Categorical Categorical Categorical Numerical Numerical Response Variable: Fitness of the final test ordering (CE score) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 8 / 15

  54. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design | T | |R ( T ) | Name CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Several applications and test suites - coverage reports derived from call-tree based adequacy (McMaster and Memon ICSM 2005) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 9 / 15

  55. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design | T | |R ( T ) | Name CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Several applications and test suites - coverage reports derived from call-tree based adequacy (McMaster and Memon ICSM 2005) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 9 / 15

  56. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design | T | |R ( T ) | Name CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Several applications and test suites - coverage reports derived from call-tree based adequacy (McMaster and Memon ICSM 2005) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 9 / 15

  57. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design | T | |R ( T ) | Name CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Use additional case study applications and adequacy criteria as future work in order to control threats to external validity Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 9 / 15

  58. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design | T | |R ( T ) | Name CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Use random and hill climbing (first and steepest ascent) as control methods for comparison to the genetic algorithm prioritizer Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 9 / 15

  59. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design | T | |R ( T ) | Name CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 See the paper for more details about the design of the empirical study (e.g., configuration of random and hill climbing prioritizers) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 9 / 15

  60. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  61. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM sel_method child_density < 0.875 0.9674 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  62. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM sel_method child_density < 0.875 0.9674 sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  63. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM sel_method child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  64. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM sel_method child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR The sel_method variable is always the most important parameter Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  65. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM sel_method child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR Importance of sel_method holds for all case study applications Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  66. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance sel_method: TOU3, TOU4, TOU5 RM sel_method child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR How does the selection method impact the genetic algorithm? Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 10 / 15

  67. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Except for the smallest application (LF), the CE scores of the evolved orderings are better than the initial and reverse test suites Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  68. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Study a type of operator as it increases in intensity , or the change in average fitness due to selection (Blickle & Thiele, Evol Comp 1997) Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  69. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Increasing selection intensity improves the CE scores of test orderings, even though it does not cause more rapid convergence Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  70. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Increasing selection intensity improves the CE scores of test orderings, even though it does not cause more rapid convergence Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  71. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Increasing selection intensity improves the CE scores of test orderings, even though it does not cause more rapid convergence Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  72. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Low intensity selection causes search to meander around low quality test suite prioritizations, making fitness stagnate and the GA terminate Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  73. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 High intensity selection focuses on a local optimum of high quality instead of hunting for hard-to-find global optimum Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  74. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 One Explanation: the fitness landscape for coverage effectiveness contains many local optima that are good test orderings Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 11 / 15

  75. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Random GB 1.0 Coverage Effectiveness (CE) 0.8 0.6 0.4 0.2 0.0 50 500 5000 Number of Random Prioritizations GB: Random orderings have average CE scores around 0.6 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 12 / 15

  76. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Random SK 1.0 Coverage Effectiveness (CE) 0.8 0.6 0.4 0.2 50 500 5000 Number of Random Prioritizations SK: Random orderings have average CE scores around 0.7 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 12 / 15

  77. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Random SK 1.0 Coverage Effectiveness (CE) 0.8 0.6 0.4 0.2 50 500 5000 Number of Random Prioritizations Conclusion: Random is not as effective as the genetic algorithm Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 12 / 15

  78. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Hill Climbing HC−FA−FN 1.0 Coverage Effectiveness (CE) 0.8 0.6 0.4 0.2 DS GB JD LF RM RP SK TM Case Study Application First Ascent: Across all applications, average CE score below 0.8 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization 13 / 15

Recommend


More recommend