Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Recent Research on Search Based Software Testing: Part 2 Francisco Chicano University of Málaga, Spain (assistant professor) Colorado State University, USA (faculty affiliate) Cádiz, Spain, July 2 nd , 2015 1 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results F. Arito et al., SSBSE 2012 Cádiz, Spain, July 2 nd , 2015 2 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Test Suite Minimization Given: Ø A set of test cases T = {t 1 , t 2 , ..., t n } Ø A set of program elements to be covered (e.g., branches) E= {e 1 , e 2 , ..., e k } Ø A coverage matrix × e 1 e 2 e 3 ... e k t 1 1 0 1 … 1 ( 1 if element e j is covered by test t i m ij = t 2 0 0 1 … 0 M= 0 otherwise … … … … … … t n 1 1 0 … 0 Find a subset of tests X ⊆ T maximizing coverage and minimizing the testing cost n X cost ( X ) = minimize c i Yoo & Harman i =1 ti ∈ X cov ( X ) = |{ e j ∈ E| ∃ t i ∈ X with m ij = 1 }| maximize Cádiz, Spain, July 2 nd , 2015 3 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results NP-hard Problems In many papers we can read … “Our optimization problem is NP-hard, and for this reason we use … • Metaheuristic techniques • Heuristic algorithms • Stochastic algorithms … which do not ensure an optimal solution but they are able to find good solutions in a reasonable time.” As far as we know: no efficient (polynomial time) algorithm exists for solving NP-hard problems But we know “inefficient” algorithms (exponential time in the worst case) Cádiz, Spain, July 2 nd , 2015 4 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results The SATisfiability Problem Can we find an assignment of boolean values (true and false) to the variables such that all the formulas are satisfied? The first NP-complete problem (Stephen Cook, 1971) If it can be solved efficiently (polynomial time) then P=NP The known algorithms solve this problem in exponential time (worst case) State-of-the-art algorithms in SAT Nowadays, SAT solvers can solve instances with 500 000 boolean variables This means a search space of 2 500 000 ≈ 10 150514 Cádiz, Spain, July 2 nd , 2015 5 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results The SATisfiability Problem Main research question: Can we use the advances of SAT solvers to solve optimization algorithms up to optimality? My favourite SAT Use SAT Optimal problem instance solvers solution Experimental Test Suite Translation to SAT MiniSAT+ Minimization Results Algorithms Cádiz, Spain, July 2 nd , 2015 6 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Outline Original PB SAT TSM Constraints Instance Instance Cádiz, Spain, July 2 nd , 2015 7 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Pseudo-Boolean Constraints A Pseudo-Boolean (PB) constraint has the form: n X a i x i � B i =1 X where where � 2 { <, , = , 6 = , >, � } , d x i 2 { 0 , 1 } . } , a i , B 2 Z , Can be translated to SAT instances (usually efficient) Are a higher level formalism to specify a decision problem Can be the input for MiniSAT+ Cádiz, Spain, July 2 nd , 2015 8 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Translating Optimization to Decision Problems Let us assume we want to minimize f(x) Check Check Check Check f ( x ) ≤ B f ( x ) ≤ B f ( x ) ≤ B f ( x ) ≤ B no no yes no B B B B Optimal solution found The same can be done with multi-objective problems, but we need more PB constraints … f m ( y ) B m f 2 ( y ) B 2 f 1 ( y ) B 1 . f ( y ) B Cádiz, Spain, July 2 nd , 2015 9 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results PB Constraints for the TSM Problem × e 1 e 2 e 3 ... e m t 1 1 0 1 … 1 ( 1 if element e j is covered by test t i m ij = t 2 0 0 1 … 0 M= 0 otherwise … … … … … … t n 1 1 0 … 0 n X m ij t i ≤ n · e j 1 ≤ j ≤ m. e j ≤ i =1 X Coverage Cost m n X X c i t i ≤ B, e j ≥ P, i =1 j =1 Cádiz, Spain, July 2 nd , 2015 10 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Example { } e 1 e 2 e 3 e 4 t 1 + t 2 + t 4 + t 5 ≤ 6 e 1 e 1 ≤ t 1 1 0 1 0 t 2 + t 6 ≤ 6 e 2 e 2 ≤ t 2 1 1 0 0 e 3 ≤ t 1 + t 3 + t 6 ≤ 6 e 3 Bi-objective problem t 3 0 0 1 0 ≤ 6 e 4 e 4 ≤ t 5 t 4 1 0 0 0 t 5 1 0 0 1 t 1 + t 2 + t 3 + t 4 + t 5 + t 6 ≤ B t 6 0 1 1 0 e 1 + e 2 + e 3 + e 4 ≥ P t 1 + t 2 + t 4 + t 5 ≥ 1 t 2 + t 6 ≥ 1 t 1 + t 3 + t 6 ≥ 1 Single-objective problem (total coverage) t 5 ≥ 1 t 1 + t 2 + t 3 + t 4 + t 5 + t 6 ≤ B Cádiz, Spain, July 2 nd , 2015 11 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Algorithm for Solving the 2-obj TSM Total coverage With coverage=|E| increase cost until success Coverage Decrease cost and find the maximum coverage again and again Cost Cádiz, Spain, July 2 nd , 2015 12 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results TSM Instances Instances from the Software-artifact Infrastructure Repository (SIR) http://sir.unl.edu/portal/index.php Table 1: Details of the inst Tests Instance Original Size e Elements to cover 4130 195 printtokens 4115 192 printtokens2 5542 208 replace 2650 126 schedule 2710 119 schedule2 1608 54 tcas 1052 117 totinfo Cost of each test: 1 Cádiz, Spain, July 2 nd , 2015 13 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Pareto Front Pareto front 100 schedule2 97.5 schedule printtokens2 printtokens 95 Table 1: Details of the inst io Algorithm Tests Original (s) R Time Instance Original Size e Elements to cover totinfo 4.61 3400.74 4130 195 printtokens 92.5 4.61 3370.44 4115 192 printtokens2 coverage level 5542 4.62 208 1469272.00 replace 2.19 492.38 2650 126 schedule 90 4.61 195.55 2710 119 replace schedule2 4.61 73.44 1608 54 tcas 4.53 181823.50 1052 117 totinfo 87.5 85 printtokens printtokens2 replace schedule 82.5 schedule2 tcas tcas totinfo 80 0 1 2 3 4 5 6 7 8 9 number of test cases Cádiz, Spain, July 2 nd , 2015 14 / 50
Test Suite Software Product Pairwise Prioritized Minimization Lines Testing in SPL Problem Formulation Landscape Theory Decomposition SAT Transf. Results Pareto Front Table 2: Pareto optimal set and Front for the instances of SIR. Instance Elements Tests Coverage Solution 195 5 100% ( t 2222 , t 2375 , t 3438 , t 4100 , t 4101 ) printtokens 194 4 99.48% ( t 1908 , t 2375 , t 4099 , t 4101 ) 192 3 98.46% ( t 1658 , t 2363 , t 4072 ) 190 2 97.43% ( t 1658 , t 3669 ) 186 1 95.38% ( t 2597 ) printtokens2 192 4 100% ( t 2521 , t 2526 , t 4085 , t 4088 ) 190 3 98.95% ( t 457 , t 3717 , t 4098 ) 188 2 97.91% ( t 2190 , t 3282 ) 184 1 95.83% ( t 3717 ) 208 8 100% ( t 306 , t 410 , t 653 , t 1279 , t 1301 , t 3134 , t 4057 , t 4328 ) replace 207 7 99.51% ( t 309 , t 358 , t 653 , t 776 , t 1279 , t 1795 , t 3248 ) 206 6 99.03% ( t 275 , t 290 , t 1279 , t 1938 , t 2723 , t 2785 ) 205 5 98.55% ( t 426 , t 1279 , t 1898 , t 2875 , t 3324 ) 203 4 97.59% ( t 298 , t 653 , t 3324 , t 5054 ) 200 3 96.15% ( t 2723 , t 2901 , t 3324 ) 195 2 93.75% ( t 358 , t 5387 ) 187 1 89.90% ( t 358 ) 126 3 100% ( t 1403 , t 1559 , t 1564 ) schedule 124 2 98.41% ( t 1570 , t 1595 ) 122 1 96.82% ( t 1572 ) 119 4 100% ( t 2226 , t 2458 , t 2462 , t 2681 ) schedule2 118 3 99.15% ( t 101 , t 1406 , t 2516 ) 117 2 98.31% ( t 2461 , t 2710 ) 116 1 97.47% ( t 1584 ) 54 4 100% ( t 5 , t 1191 , t 1229 , t 1608 ) tcas 53 3 98.14% ( t 13 , t 25 , t 1581 ) 50 2 92.59% ( t 72 , t 1584 ) 44 1 81.48% ( t 217 ) 117 5 100% ( t 62 , t 118 , t 218 , t 1000 , t 1038 ) totinfo 115 4 98.29% ( t 62 , t 118 , t 913 , t 1016 ) 113 3 96.58% ( t 65 , t 216 , t 913 ) 111 2 94.87% ( t 65 , t 919 ) 110 1 94.01% ( t 179 ) Cádiz, Spain, July 2 nd , 2015 15 / 50
Recommend
More recommend
