Predicting Fault Numbers via Testing Marc Roper Dept. Computer and - PowerPoint PPT Presentation

Predicting Fault Numbers via Testing Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 1 / 12

Motivation Even after testing there are still going to be faults in software - how do we estimate how many are left? Useful for: Reliability estimation Decision making Release? Test further? Automated debugging etc... Aim of this talk is to look at the applicaiton of capture-recapture techniques to generate predictions of faults remaining after testing Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 2 / 12

Capture-Recapture Techniques Used by population ecologists to estimate number of particular species in an area Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 3 / 12

Capture-Recapture Techniques Used by population ecologists to estimate number of particular species in an area Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 3 / 12

Capture-Recapture Techniques Used by population ecologists to estimate Number of iterations is number of particular species in an area variable: animals may get caught 0 - many times Data fed into capture-recapture models to make predictions about total population in the area Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 3 / 12

Mapping Capture-Recapture Techniques to Software Testing Animal = Fault Capture Method = Testing Technique Trapping Occasion = Independent Application of a Testing Technique (just 2 considered) Number of models explored which incorporate different assumptions: M t Probability of fault detection differs between techniques/testers M h Probability of fault detection differs between faults M th Combines the above so that both faults and techniques/testers become sources of variation Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 4 / 12

Mapping Capture-Recapture Techniques to Software Testing Various estimators for these models, e.g. Lincoln-Petersen estimator for M t ˆ N = n 1 n 2 m 1 Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 5 / 12

Mapping Capture-Recapture Techniques to Software Testing Various estimators for these models, e.g. Lincoln-Petersen estimator for M t ˆ N = n 1 n 2 m 1 Five considered in total: LP Lincoln-Peterson estimator for M t JK Jackknife estimator for M h CMT Chao estimator for M t CMH Chao estimator for M h CMTH Chao estimator for M th Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 5 / 12

Experimental Evaluation Data: System Size (loc) Total Faults No. Testers Testing Strategy Strathclyde1 ˜200 8 47 Functional, Structural Strathclyde2 ˜200 9 47 Functional, Structural Strathclyde3 ˜200 8 47 Functional, Structural Myers 63 15 30 ∗ Functional, Structural Space 6218 38 30 ∗∗ Statement coverage + * split into 2 groups of 15 ** simulated, split into 3 groups of 10 Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 6 / 12

Experimental Evaluation Data: System Size (loc) Total Faults No. Testers Testing Strategy Strathclyde1 ˜200 8 47 Functional, Structural Strathclyde2 ˜200 9 47 Functional, Structural Strathclyde3 ˜200 8 47 Functional, Structural Myers 63 15 30 ∗ Functional, Structural Space 6218 38 30 ∗∗ Statement coverage + * split into 2 groups of 15 ** simulated, split into 3 groups of 10 Method: Each of the five estimators evaluated using fault data revealed by each tester Every possible pairwise combination considered Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 6 / 12

Results (Strathclyde) Values of Estimators for Program2 (N=9): 20 48 29 62 40 36 65 18 62 29 48 16 30 88 58 62 56 14 34 20 2 79 34 20 58 62 56 12 56 56 58 62 10 62 58 59 38 47 60 63 28 5 8 8 6 54 50 64 68 17 45 51 51 4 74 2 0 N = 91 91 91 91 91 91 91 91 91 91 91 91 91 91 91 LPFTP2 LPBTP2 JKSTP2 CMTFTP2 CMTBTP2 CMHSTP2 CMTHFTP2 CMTHBTP2 LPSTP2 JKFTP2 JKBTP2 CMTSTP2 CMHFTP2 CMHBTP2 CMTHSTP2 Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 7 / 12

Results (Myers) Values of Estimators for Myers Data (N=15) 30 77 55 110 33 77 25 25 106 81 71 114 110 33 3 20 20 70 54 15 10 5 118 82 0 N = 95 95 95 95 95 95 95 95 95 95 95 95 95 95 95 LPFT LPBT JKST CMTFT CMTBT CMHST CMTHFT CMTHBT LPST JKFT JKBT CMTST CMHFT CMHBT CMTHST Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 8 / 12

Results (Space) Values of Estimators for Space Data (N=38) Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 9 / 12 MS Min-statement

Impact of Coverage Data Strathclyde data scaled according to coverage (N=9) 20 58 30 48 62 18 58 16 34 20 56 62 40 29 56 14 34 20 28 47 59 58 56 58 47 28 59 12 10 8 6 4 2 0 N = 91 91 91 91 91 91 91 91 91 91 LPSTP2 LPBTP2 JKSTP2 JKBTP2 CMTSTP2 CMTBTP2 CMHSTP2 CMHBTP2 CMTHSTP2 CMTHBTP2 Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 10 / 12

Impact of Coverage Data SPACE data scaled according to coverage (N=38) Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 11 / 12

Conclusions Capture-recapture techniques show promise as predictors of faults numbers Performance of the models tends to vary between datasets Findings illustrate the estimators’ susceptibility to the data being used – in particular the pattern of overlap and distinctiveness between faults Diversity generated by different testers using different testing approaches are likely to generate the most accurate results However, too much diversity with respect to the overlap amongst faults may also lead to inaccurate estimates. Future work will look at more datasets and how random or evolved data can be used to improve the accuracy of estimates Marc Roper Dept. Computer and Information Sciences University of Strathclyde Glasgow, U.K. Predicting Fault Numbers via Testing 12 / 12