Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary - PowerPoint PPT Presentation

Introduction Resource-Constrained Testing Conclusions and Future Work Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results Suvarshi Bhadra and Gregory M. Kapfhammer Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/ TAIC PART 2008, Fast Abstract Track, August 2008 Featuring images from www.pdclipart.org Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 1 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Important Contributions sss − covers ssm − covers ssl − covers 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 Overlap Score Overlap Score Overlap Score 0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 0.0 NCO JCO TCO NCO JCO TCO NCO JCO TCO Overlap Metric Overlap Metric Overlap Metric Technique Formulation Empirical Results Design , implement and empirically evaluate test suite prioritizers that leverage travelling salesperson problem (TSP) solvers to efficiently find cost-effective orderings Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 2 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Regression Testing Techniques Before After Before After Reduction Prunes the Test Suite Prioritization Reorders the Tests It may expensive to run a test suite T = � T 1 , . . . , T n � . Prioritization searches through the n ! = n × n − 1 × . . . × 1 orderings for those that avoid costly database restarts , Web service calls , or memory interactions . Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 3 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Prioritizing When Memory is Constrained Frequent Memory Rewrites High Testing Costs Frequent reads and writes to memory may increase execution time by as much as 600% when a Java application executes on a virtual machine with a small heap Solution: maximize memory reuse between test cases Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 4 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work The Impact of Test Ordering m 1 m 2 m 3 m 4 m 5 m 6 Test 30 30 30 30 30 30 Size T 1 90 • • • T 2 90 • • • T 3 90 • • • T 4 90 • • • T 5 60 • • T = � T 1 , T 2 , T 3 , T 4 , T 5 � transfers 750 units to and from memory T ′ = � T 2 , T 4 , T 1 , T 3 , T 5 � only loads and unloads 180 units Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 5 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Test Prioritization: Steps One and Two Program 256 N 3 N 4 553 Adequacy Criterion 256 553 553 553 256 256 256 553 Residual Test Instrumented Instrumentation Requirements Program 553 N 2 256 234 N 1 553 256 678 256 256 553 234 Test Suite O 553 553 Execution 256 0 256 256 256 234 N 5 N 0 Cummulative Covered 256 Adequacy Calculator Requirements Collect method invocation and Formulate a complete graph using equations that estimate size data using test coverage monitor and profiler costs for all test pairs Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 6 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Test Prioritization: Steps Three and Four 2.5 Execution Time (sec) 2.0 1.5 1.0 10 20 30 40 50 60 70 80 90 100 Size of the Test Suite (n) Evaluate the efficiency of the TSP Use TSP solvers to identify a solvers and the effectiveness of Hamiltonian path with low the test orderings estimated costs Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 7 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Empirical Results Time v No of Test Cases Score v No of Test Cases − NN 100 0.20 90 0.15 80 (score) (t) 0.10 70 NN RNN 60 NI 0.05 FI CI 50 AI 0.00 10 20 30 40 50 0 200 400 600 800 1000 (n) (n) High Percentile Rankings Efficient Prioritizers Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 8 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Avoiding Database Restarts T2 T16 T12 T3 T1 T8 T0 T7 T5 T13 T14 T4 T10 T11 T15 T6 T9 Use prioritization to avoid costly database restarts Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 9 / 10

Introduction Resource-Constrained Testing Conclusions and Future Work Concluding Remarks sss − covers ssm − covers ssl − covers 1.0 1.0 1.0 Program Reduction 0.8 0.8 0.8 Analysis 0.6 0.6 0.6 Overlap Score Overlap Score Overlap Score Technique 0.4 0.4 0.4 0.2 0.2 0.2 Test Suite Prioritization 0.0 0.0 0.0 NCO JCO TCO NCO JCO TCO NCO JCO TCO Overlap Metric Overlap Metric Overlap Metric Initial Empirical Evaluation Test Suite Prioritizers Preliminary results with synthetic test suites indicate that it is possible to prioritize test suites with TSP solvers Use different methods for solving TSP instances (e.g., order-based genetic algorithms) and include real-world applications http://www.cs.allegheny.edu/~gkapfham/research/ Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results 10 / 10