Software Testing with Evolutionary Strategies Introduction - PowerPoint PPT Presentation

Rapid Integration of Software Engineering Techniques, 2005 1/22 Software Testing with Evolutionary Strategies Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work Enrique Alba and J. Francisco Chicano Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 2/22 Introduction • After codification, software products require a test phase • The objective is to find errors and to ensure software correctness Introduction • Software companies dedicate 50% of resources to this task Previous Work 1.0, 2.3 1.0, 2.3 Test Data Generator 10.5 Ok! Evolutionary Strategy • We propose an automatic tool based on Metaheuristics to generate the input data for the test Experiments 2.7, 5.4 Conclusions & Wrong! 2.7, 5.4 Future Work 15.0 Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 3/22 Test Adequacy Criteria • Test data generator objective: to propose input data finding a maximum number of errors Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 4/22 Test Adequacy Criteria • Test data generator objective: to propose input data finding a DIFFICULT TO CHECK maximum number of errors Introduction • Test data generator objectives (test adequacy criteria) Previous Work Test Data Generator Evolutionary Statement Coverage All the statements executed Strategy Branch Coverage All the branches taken Experiments Conclusions & Condition-Decision All the atomic conditions Future Work Coverage true and false Stronger Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 5/22 Previous Work • Three main paradigms: � Random test data generation Introduction 1.2, 0.7 Inputs: a,b Previous Work Test Data c = a+b � Symbolic test data generation Generator α , β Evolutionary α + β < 0 1.0, -2.0 true false Strategy c < 0 α + β ≥ 0 3.5, 1.2 Experiments � Dynamic test data generation Conclusions & -1.0, -0.5 Future Work 1.0, -0.5 Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 6/22 Previous Work • The global objective is broken down in small sub-objectives Six sub-objectives Introduction condition-decision Previous Work coverage c1 true c1 false c2 true c2 false Test Data Generator c3 true c3 false Evolutionary Strategy Function minimization problem Experiments Current input data Distance Conclusions & Future Work Sub-objective (c3 true) Input data Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 7/22 Test Data Generator Test Data Generator Introduction Select sub-objective Previous Work Input data Evolutionary Test Data Test Program Strategy Fitness Generator Evolutionary Strategy yes Continue? Experiments no Conclusions & Future Work End Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 8/22 Evolutionary Strategy Evolutionary Algorithms Introduction Evolutionary Genetic Strategy Algorithm Previous Work Genetic Evolutionary Test Data Generator Programming Programming Evolutionary Strategy Population-based Experiments Conclusions & Future Work Optimization problem Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 9/22 Evolutionary Strategy • Pseudo-code of an Evolutionary Strategy Introduction t := 0; P(0) := Generate(); Previous Work Evaluate (P(0)); while stop criterion not met do Test Data P’(t) := Select (P(t)); Generator P’’(t) := Gaussian_Mutation (P’(t)); Evaluate(P’’(t)); Evolutionary P(t+1) := Replacement (P(t), P’’(t)); Strategy t := t+1; Experiments end while; return the best solution; Conclusions & Future Work Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 10/22 Evolutionary Strategy • Evolutionary Strategy � Individual Introduction → Solution vector (0.2, -1.4, 3.5) (1.0, 10.3, 7.2) → Standard deviations Previous Work (1.7, 0.3, 2.1) → Angles Test Data � Gaussian Mutation Generator Selected solution Evolutionary Strategy New solution Experiments New population Conclusions & � Replacement → ( μ + λ ) and ( μ , λ ) Future Work μ individuals Previous population New individuals Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 11/22 Evolutionary Strategy Population Test Program Introduction μ individuals Previous Work Test Data Generator Gaussian Mutation Evolutionary (0.1, -5.2, 3.0, ...) Strategy 7.3 λ individuals Experiments Conclusions & ( μ + λ ) Replacement Future Work New population Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 12/22 Experiments: Benchmark • Benchmark: 11 programs in C language Program Conds. LOC Args. Source Introduction triangle 21 53 3 Michael et al., 2001 gcd 5 38 2 Authors Previous Work calday 11 72 3 C-Recipes crc 9 82 13 C-Recipes Test Data insertion 5 47 20 C-Recipes Generator shell 7 58 20 C-Recipes Evolutionary quicksort 18 143 20 C-Recipes Strategy heapsort 10 72 20 C-Recipes select 28 200 21 C-Recipes Experiments bessel 21 245 2 C-Recipes Conclusions & sa 30 332 23 C-Recipes Future Work • We perform 30 independent runs to get statistical confidence Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 13/22 Experiments: First Results • Algorithm: (10+1)-ES Program Avg. Avg. Avg. • Maximum iterations: 100 Cov.(%) Evals. Time(s) Introduction while(1<2) triangle 97.54 1975 10.6 { Previous Work gcd 100.00 21 0.0 /* something */ } calday 91.82 1182 6.8 Test Data crc 94.44 1114 28.7 Code-dependent coverage loss Generator insertion 100.00 10 0.0 Evolutionary shell 100.00 10 0.0 p = malloc(4); Strategy if (!p) quicksort 88.89 1110 8.4 { heapsort 90.00 1110 9.4 error(); Experiments select 53.57 1120 8.9 } Conclusions & Environment-dependent bessel 95.08 1306 7.6 Future Work coverage loss sa 97.06 1329 5082.1 • Condition-decision is not a good metric for measuring accuracy Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 14/22 Experiments: First Results reached sub-objectives Corrected coverage = reachable sub-objectives Introduction (10+1)-ES Corrected Coverage (%) Program Cov.(%) Avg.(%) Max(%) Std. Dev. Previous Work triangle 97.54 99.92 100.00 0.0044 Test Data gcd 100.00 100.00 100.00 0.0000 Generator calday 91.82 91.82 100.00 0.0246 crc 94.44 100.00 100.00 0.0000 Evolutionary insertion 100.00 100.00 100.00 0.0000 Strategy shell 100.00 100.00 100.00 0.0000 Experiments quicksort 88.89 94.12 94.12 0.0000 heapsort 90.00 100.00 100.00 0.0000 Conclusions & select 53.57 83.33 83.33 0.0000 Future Work bessel 95.08 97.40 97.56 0.0061 sa 97.06 98.70 100.00 0.0072 Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 15/22 Experiments: Influence of λ Number of offspring: 1 2 3 100 90 Introduction 80 Corrected Coverage (%) Previous Work 70 60 Test Data 50 Generator 40 30 Evolutionary Strategy 20 10 Experiments 0 triangle gcd calday crc insertion shell quicksort heapsort select bessel sa Test Programs Conclusions & Future Work Coverage: negligible Influence of λ Number of evaluations: important Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 16/22 Experiments: Influence of λ Number of offspring: Number of offspring: 1 2 3 1 2 3 100 4500 90 4000 Introduction 80 3500 Corrected Coverage (%) Previous Work 70 3000 Evaluations 60 2500 Test Data 50 Generator 2000 40 1500 30 Evolutionary 1000 Strategy 20 500 10 Experiments 0 0 triangle gcd calday crc insertion shell quicksort heapsort select bessel sa triangle gcd calday crc insertion shell quicksort heapsort select bessel sa Test Programs Test Programs Conclusions & Future Work Coverage: negligible Influence of λ Number of evaluations: important Heraklion Crete, Greece, September 8-9, 2005

Rapid Integration of Software Engineering Techniques, 2005 17/22 Experiments: Influence of μ Population size: 1 5 10 20 30 100 90 Introduction 80 Corrected Coverage (%) 70 Previous Work 60 Test Data 50 Generator 40 30 Evolutionary 20 Strategy 10 0 Experiments triangle gcd calday crc insertion shell quicksort heapsort select bessel sa Test Programs Conclusions & Future Work Coverage: negligible for μ ≥ 5 Influence of μ Number of evaluations: slight Heraklion Crete, Greece, September 8-9, 2005