Faults Found by Random Testing WanChi Chio Justin Molinyawe
Introduction ● Why apply random testing to object oriented software? ○ Cost-efficient in terms of implementation effort and execution time.
Introduction (Cont.) ● Characteristics of random testing ○ Predictability in terms of number and nature of faults. ○ Average time it takes to find a failure. ○ Distribution of detected faults.
Introduction (Cont.) Experiments ● Predictability of Random Testing ○ Creating and using tests with AutoTest for 27 classes from Eiffel library. ● Each class was tested for 90 minutes. For each class, testing sessions were ran 30 times with different seeds.
Unit Tests for Object- Oriented Programs ● Purpose: to verify the run of a particular routine: ○ test routine "test_x" for routine "x" ● Hard-coded stubs must be used to replace non-trivial routines which are not under test for a strict interpretation, but it does not usually happens in practice
Unit Tests for Object-Oriented Programs (Cont.) ● Input are created in two ways by: ○ loading previously saved objects into memory => to provide complete control over the input creation ○ calling a series of constructors to create and modify data through normal means => to yield more relevant data
Unit Tests for Object-Oriented Programs (Cont.) ○ Oracle/expect output can be provided in different levels of abstractions for validation through following specifications: ■ The exact expected output - good for complex scenarios ■ Conditions that the output should fulfill for any input ■ "No exception" as the expected output - easier to be re-used for other tests
Random Tests Cases for Eiffel Library Experiment ● Testing is completely automated with test cases created under the following criterias: ○ Unit - routines of Eiffel classes ■ Test synthesizer makes test cases where the goal is to test each particular routine (Test Case Synthesis Algorithm) ○ Input - Objects are created through regular constructor and routine calls. ■ To enforce diversity, the initial objects are modified for new test cases
Random Tests Cases for Eiffel Library Experiment (Cont.) ○ Oracle ■ Contract-based by containing specifications in the form of routine pre and postconditions and class invariants
Faults and failures of Random Testing ● Usually, failures are an observed difference between the actual and intended behavior. ● Results relied on an approximation that maps failures to faults based on: ○ Two failures are a consequence of the same fault if and only if they manifest themselves through the same type of exception.
Test Case Synthesis Algorithm Class C { R m(C 1 P 1 , ... , C n P n ); } A test case for routine m above can be executed as follows: c.m(a 1 , .... , a n ) => r where c = object of Class C r = output in Type R m = routine defined in Class C a j = argument object of Type C j for Parameter P j
Test Case Synthesis Algorithm (Cont.)
Using AutoTest ● Contract based testing tool. ○ Allows easy plug-in of testing strategies. ● Composed of two processes ○ Test Generator / Driver ○ Interpreter ● AutoTest runs for a duration of the set time.
Number of Faults ● Will different runs of random testing with different seeds for the pseudo-random number generator reveal a similar number of faults? ● Will the faults be the same?
Number of Faults in Eiffel Library Experimental Setup ● 27 classes ● Each class was tested for 30 sessions of 90 mins ● Each session had a different seed to initialize the pseudo-random number generator. ● Total testing time was 30 * 27 * 90 = 50.6 days
Number of Faults in Eiffel Library Experimental Results ● Is random testing predictable? Two kinds of predictability: ○ Number of faults ○ Identity of faults and find if it is likely to detect the same faults regardless of the experiments was selected ● Normalized results give us aggregated results.
Number of Faults
Conclusion Results ● On average, 30% of the faults found during experiments are found after 10 mins. After 90 mins, 8% points, on average of the overall number of randomly detected faults are found. ● Relative to the number of detected faults, random+ is highly predictable. ● Different runs of the testing process reveal different faults.
Questions?
Recommend
More recommend
