The Theory and Practice of Software Testing: Can we Test it? Yes we - PowerPoint PPT Presentation

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion The Theory and Practice of Software Testing: Can we Test it? Yes we Can! Gregory M. Kapfhammer † Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/ SGT Global, February 2008 † In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS) Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS) Featuring images from Embroidery and Tapestry Weaving , Grace Christie (Project Gutenberg) The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 1 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Presentation Outline Software Testing Challenges 1 Structural Testing 2 Regression Testing 3 Mutation Testing 4 Future Work 5 Conclusion 6 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 2 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion The Challenge of Software Testing I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra , Communications of the ACM, 1968 Important Question : What are your software development and testing challenges ? What are your best solutions ? The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 3 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Modern Software is Complex Complex source code, database, files, and network communication Can we increase reliability by simplifying software? The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 4 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Defect Locations P Input Output a 5 print ... exit Final Result: 45 Byte Code Virtual Graphical Machine Database Interface File Operating System System Defects may exist in the individual components or the interactions The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 5 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Approaches to Software Testing Structural Testing Mutation Testing Regression Specification Testing Testing Testing isolates defects and establishes a confidence in the correctness of a software application The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 6 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Presentation Outline Software Testing Challenges 1 Structural Testing 2 Regression Testing 3 Mutation Testing 4 Future Work 5 Conclusion 6 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 7 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion What is a Test Case? Expected Output Set Up Method Test Input Output Verdict Under Test Oracle Tear Down Overview Test suite executor (JUnit) runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 8 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Test Coverage Monitoring Program Adequacy Criterion Residual Test Instrumented Instrumentation Requirements Program Test Suite Execution Cummulative Covered Adequacy Calculator Requirements Overview Structural adequacy criteria focus on the coverage of nodes, edges, paths, and definition-use associations Instrumentation probes track the coverage of test requirements The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 9 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Calculating the Coverage of a Test Test Suite Calculating Coverage Adequacy Criterion Use instrumentation probes to capture and analyze a Program Instrumented Program test suite’s coverage of the Instrumentation program Instrumented Test Suite Regression Testing Test Coverage Coverage Database Monitoring Results The adequacy Test Adequacy measurements can be used Requirements Calculation to support both test suite Adequacy reduction and prioritization Measurements The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 10 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Presentation Outline Software Testing Challenges 1 Structural Testing 2 Regression Testing 3 Mutation Testing 4 Future Work 5 Conclusion 6 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 11 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Finding the Overlap in Coverage R R 2 R 4 R 6 R 7 R 5 R 1 R 3 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements May include the remaining tests so that they can redundantly cover the requirements The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 12 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Reducing and Prioritizing the Tests Original Test Suite GRT Repeat Reduction Modified Test Suite Begin Coverage Report Testing Results End or Prioritization Test Suite Execution Program VSRT Repeat Regression Testing Overview Reduction creates a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively. Techniques use heuristics to solve NP-complete problems. The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 13 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Evaluating a Test Prioritization Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [ 0 , 1 ] The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 14 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Characterizing a Test Suite Test Information Test Case Cost (sec) Requirements R 1 R 2 R 3 R 4 R 5 T 1 5 � � T 2 10 � � � � T 3 4 � � � Total Testing Time = 19 seconds Formulating the Metrics CE considers the execution time of each test while CE u assumes that all test cases execute for a unit cost The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 15 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Coverage Effectiveness Values Calculating CE and CE u Ordering CE CE u T 1 T 2 T 3 . 3789 . 4 T 1 T 3 T 2 . 5053 . 4 T 2 T 1 T 3 . 3789 . 5333 T 2 T 3 T 1 . 4316 . 6 T 3 T 1 T 2 . 5789 . 4557 T 3 T 2 T 1 . 5789 . 5333 Observations Including test case costs does impact the CE metric Depending upon the characteristics of the test suite, we may see CE = CE u , CE > CE u , or CE < CE u The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 16 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Presentation Outline Software Testing Challenges 1 Structural Testing 2 Regression Testing 3 Mutation Testing 4 Future Work 5 Conclusion 6 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 17 / 23

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion Mutation Testing Techniques Program Mutant Creation A mutation testing tool (e.g., Mutant µ Java or Jumble) inserts defects Generator into the program under test Question : Why are we inserting Mutant faults into the the programs that we Programs are testing? Test Suite Test Quality Execution Goal : measure the quality of the test suite by determining whether or not it Mutation can differentiate between faulty and Score non-faulty programs The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 18 / 23