A Theoretical & Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA July 2007 Mark Harman Phil McMinn Sheffield University King’s College London Mark Harman ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Which Search Technique: Global or Local? Mark Harman Phil McMinn Sheffield University King’s College London Mark Harman ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Which Search Technique: Global or Local? Mark Harman and Phil McMinn. A Theoretical and Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA 2007. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search TSE. To appear. Mark Harman ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Which Search Technique: Global or Local? Mark Harman and Phil McMinn. A Theoretical and Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA 2007. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search TSE. To appear. Mark Harman ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Which Search Technique: Global or Local? Mark Harman and Phil McMinn. A Theoretical and Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation ISSTA 2007. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search TSE. To appear. Author order is alphabetical Mark Harman ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Where is King’s College London? UCL LSE King’s London Eye IC Mark Harman 6 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Where is Sheffield University? Sheffield London Mark Harman 7 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
No Full Monty Joke Mark Harman 8 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
No Full Monty Joke Sorry Mark Harman 9 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Overview Search Based Testing Local : Hill Climbing using Alternating variable method Global : Genetic Algorithms Theoretical foundations Schemas Royal Roads Empirical study Implications Mark Harman 10 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
What is SBT In Search based testing we apply search techniques to search large input spaces, guided by a fitness function. Mark Harman 11 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
What is SBT In Search based testing we apply search techniques to search large input spaces, guided by a fitness function. Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization Mark Harman 12 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
What is SBT In Search based testing we apply search techniques to search large input spaces, guided by a fitness function. Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization Mark Harman 13 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
What is SBT In Search based testing we apply search techniques to search large input spaces, guided by a fitness function. Genetic Algorithms, Hill climbing, Simulated Annealing, Random, Tabu Search, Estimation of Distribution Algorithms, Particle Swarm Optimization Mark Harman 14 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Structural Testing Focus on branch testing Most widely studied So ready for some more in depth analysis Mark Harman 15 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Other Search Based Testing Applications Temporal Wegener et al. Coverage Pargass & Harrold, Xanthakis et al., McMinn, Harman, Michael et al, Sthamer, Jones … Functional Wegener et al. Regression Rothermel et al., Woolcott et al., Yoo and Harman,… Interaction Cohen et al. Bryce, Colbourn Exception Tracey and Clark Stress Briand et al., Antoniol, Di Penta Robustness Shultz et al. Mark Harman 16 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Structural Testing Focus on branch testing Most widely studied So ready for some more in depth analysis Two algorithms: Hill Climbing, using Korel’s alternating variable method Genetic Algorithms, using DaimlerChrysler approach Mark Harman 17 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Structural Testing Focus on branch testing Most widely studied So ready for some more in depth analysis Two algorithms: Hill Climbing, using Korel’s alternating variable method Genetic Algorithms, using DaimlerChrysler approach … and Random Search Mark Harman 18 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Fitness Computation 1. Approximation level Level 4 Level 3 Identify relevant branching statements using control dependence Level 2 Evaluation of predicate in a branching condition Level 1 if A = B Local_Distance = | A - B | Target Target Target Target Fitness = Approximation_Level + Local_Distance Mark Harman 19 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Alternating Variable Method The alternating variable method is hill climbing plus accelerated moves Near Neighbour? One small increase For some input variable One small decrease Method: Cycle through input variables one at a time: probe moves move to near neighbour: If probing works, make accelerated pattern moves Until no improvement on any variable Mark Harman 20 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Goal-Oriented Approach: Alternating Variable Method Fitness Accelerated hill climb Input variable value Mark Harman 21 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Hill Climbing Steepest Descent Mark Harman 22 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Hill Climbing Steepest Descent Mark Harman 23 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Alternating Variable Method Example Random start: a=10 b=20 void example(int a, int b, int c) { c=30 if (a == 0) { ... } Case a :- Probe move has if (b == 0) { no e fg ect if (c == 0) { // target Case b :- } Decrease probe } improves } So accelerate until b=0 Case c :- Mark Harman 24 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Evolutionary Algorithms Insertion Mutation Test Fitness evaluation execution Recombination End? Selection Mark Harman 25 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Evolutionary Testing Insertion Test cases Mutation Test Fitness evaluation execution Recombination End? Selection Mark Harman 26 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Evolutionary Testing Insertion Test cases Mutation Test Execution Fitness evaluation execution Recombination End? Selection Mark Harman 27 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Evolutionary Testing Insertion Test cases Mutation Test Execution Fitness evaluation execution Recombination Monitoring End? Selection Mark Harman 28 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
How mating makes life easier Mark Harman 29 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
How mating makes life easier Mating is really very much an analogy The important property is crossover Mark Harman 30 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
How mating makes life easier Mark Harman 31 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
How mating makes life easier Cost: 3 Cost: 4 Cost: 8 Cost: 2 a & b a & b a & b a & b a & b a & b cost: 1 cost: 1 cost: 3 cost: 3 cost: 1 cost: 1 cost: 3 cost: 3 cost: 1 cost: 3 cost: 1 cost: 3 c & d c & d c & d c & d c & d c & d cost: 2 cost: 2 cost: 2 cost: 2 cost: 2 cost: 2 e & f e & f e & f e & f e & f e & f cost: 1 cost: 1 cost: 1 cost: 1 cost: 1 cost: 1 cost: 3 cost: 3 cost: 3 cost: 3 cost: 3 cost: 3 Mark Harman 32 ISSTA: Empirical and Theoretical Search Based Testing Thursday, 10 December 2009
Recommend
More recommend
