Towards Regression Testing for Database Applications Gregory M. - PowerPoint PPT Presentation

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Towards Regression Testing for Database Applications Gregory M. Kapfhammer † Department of Computer Science Allegheny College, Pennsylvania, USA http://cs.allegheny.edu/~gkapfham/ ASTReNet and SOSoRNet, King’s College London, 2007 † In Conjunction with Mary Lou Soffa (UVa/CS), Panos Chrysanthis (Pitt/CS), Bruce Childers (Pitt/CS) Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Testing Database Applications Research Contribution A regression testing framework for traditional database applications. Future research includes service-oriented applications that use Grid-enabled databases. Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions An Interesting Defect Report Database Server Crashes When you run a complex query against Microsoft SQL Server 2000, the SQL Server scheduler may stop responding. Additionally, you receive an error message that resembles the following: Date Time server Error: 17883 Severity: 1, State: 0 Date Time server Process 52:0 (94c) ... An Input-Dependent Defect This problem occurs when one or more of the following conditions are true: The query contains a U N I O N clause or a U N I O N A L L clause that affects many columns. The query contains several J O I N statements. The query has a large estimated cost. BUG 473858 (SQL Server 8.0) Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions An Interesting Defect Report Database Server Crashes When you run a complex query against Microsoft SQL Server 2000, the SQL Server scheduler may stop responding. Additionally, you receive an error message that resembles the following: Date Time server Error: 17883 Severity: 1, State: 0 Date Time server Process 52:0 (94c) ... An Input-Dependent Defect This problem occurs when one or more of the following conditions are true: The query contains a U N I O N clause or a U N I O N A L L clause that affects many columns. The query contains several J O I N statements. The query has a large estimated cost. BUG 473858 (SQL Server 8.0) Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Real World Example A Severe Defect The Risks Digest, Volume 22, Issue 64, 2003 Jeppesen reports airspace boundary problems About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20. An Important Point Practically all use of databases occurs from within application programs [Silberschatz et al., 2006, pg. 311] Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Real World Example A Severe Defect The Risks Digest, Volume 22, Issue 64, 2003 Jeppesen reports airspace boundary problems About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20. An Important Point Practically all use of databases occurs from within application programs [Silberschatz et al., 2006, pg. 311] Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Program and Database Interactions Basic Operation Program P creates SQL P m statements in order to view and/or modify the state of the relational database insert select update delete SQL Construction Static analysis does not reveal the exact SQL command since D D the program constructs the full 1 e SQL statement at run-time Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Database Interaction Granularity P R 1 A B C D m i R 2 D k E F G H R 3 m j I J K L D l Database Interactions Program P interacts with two relational databases D k and D l at different levels of granularity (relation, record, attribute, ...) Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Overview of the Coverage Monitoring Process Test Suite Calculating Coverage Adequacy Criterion Use instrumentation probes to capture and analyze a Program Instrumented Program program’s interaction with the Instrumentation databases Instrumented Test Suite Regression Testing Test Coverage Coverage Database Monitoring Results The adequacy Test Adequacy measurements can be used Requirements Calculation to support both test suite Adequacy reduction and prioritization Measurements Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Database-Aware Coverage Trees Instrumentation Probes Test Case Use static and dynamic Method Invocation (load-time) instrumentation techniques to insert coverage Database Interaction Point monitoring probes Database Coverage Trees Relation Store the coverage results in a tree in order to support the Attribute Record calculation of many types of coverage (e.g., data flow or Attribute Value call tree ) Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Comparing the Coverage Trees Tree Characteristics Tree DB? Context Probe Time Tree Space CCT × Partial Low - Moderate Low DCT × Full Low Moderate - High DI-CCT Partial Moderate Moderate � DI-DCT Full Moderate High � Table Legend Database? ∈ {× , � } Context ∈ { Partial, Full } Probe Time Overhead ∈ { Low, Moderate, High } Tree Space Overhead ∈ { Low, Moderate, High } Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Database-Aware Regression Testing Original Test Suite GRT Repeat Reduction Modified Test Suite Begin Coverage Report Testing Results End or Prioritization Test Suite Execution Program and Database VSRT Repeat Regression Testing Overview Reduction aims to find a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively. Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Finding the Overlap in Coverage R R 2 R 1 R 3 R 4 R 6 R 7 R 5 T 2 T 4 T 10 T 6 T 8 T 12 T 1 T 3 T 11 T 7 T 9 T 5 Test Suite Reduction R j → T i means that requirement R j is covered by test T i T = � T 2 , T 3 , T 6 , T 9 � cover all of the test requirements Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Measuring Coverage Effectiveness Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Configuring the Regression Testing Framework Configuration of the Regression Tester Technique Test Cost Requirements Reduction Prioritization Type Unit Actual Type Traditional Database-Aware Greedy HGS Delayed Greedy Reverse Random Data Flow Coverage Tree Path Overlap-Aware Not Overlap-Aware Unique Dominant Type of Tree Cost Coverage Ratio Super Path Containing Path DCT CCT Regression tester uses several algorithms and test requirements Gregory M. Kapfhammer Towards Regression Testing for Database Applications