Verifying Test Hypotheses - HOL/TestGen An Experiment in Test and - PowerPoint PPT Presentation

Verifying Test Hypotheses - HOL/TestGen Verifying Test Hypotheses - HOL/TestGen An Experiment in Test and Proof Thomas Malcher January 20, 2014 1 / 20

Verifying Test Hypotheses - HOL/TestGen HOL/TestGen Outline Introduction Test Hypotheses HOL/TestGen - Demo Verifying Test Hypotheses Conclusion 2 / 20

Verifying Test Hypotheses - HOL/TestGen HOL/TestGen What it does? builds test case and test data from a formal/logical test specification the Model is a logical expression (test specification) dynamic testing/blackbox testing generates functional test strategy 3 / 20

Verifying Test Hypotheses - HOL/TestGen HOL/TestGen What it is? semi-automated test tool for specification based tests Academic tool to get insight in the connection between test and verification What is the nature of the relation between test and proof? Does a test approximate a verification? Does testing contribute to verification? 4 / 20

Verifying Test Hypotheses - HOL/TestGen HOL/TestGen Foundations Isabelle/HOL Higher-Order-Logic (HOL) like a functional language extended by logical quantifiers Test Hypotheses i s s o r t e d ( [ ] ) = True 1 i s s o r t e d ( x#xs ) = ( case xs of [ ] − − > True | 2 y#ys − − > ( ( x < y ) or ( x=y ) ) and i s s o r t e d ( xs ) ) 3 5 / 20

Verifying Test Hypotheses - HOL/TestGen HOL/TestGen The Core Decompose a test specification (TS), into a semantically equivalent set of Test cases TC i and Test hypotheses H i [ TC 1 . . . TC n , H 1 . . . H m ] = ⇒ TS If the system under test passes all test cases TC i successfully, and if it satisfies all test hypothesis H i , it is correct with respect to the test specification TS. Or: validation by test + proof of test hypotheses → validation by proof 6 / 20

Verifying Test Hypotheses - HOL/TestGen Test Hypotheses[5] What is a Test Hypothesis? if we do tests we set up hypotheses we infer from the behaviour for a specific input to the behaviour for a class of inputs 7 / 20

Verifying Test Hypotheses - HOL/TestGen Test Hypotheses We can distinguish between 2 types of test hypotheses: uniformity hypotheses ∃ x : P ( x ) = ⇒ ∀ x : P ( x ) regularity hypotheses ∀ x : | x | < = k ∧ P ( x ) = ⇒ ∀ x : k < | x | ∧ P ( x ) e.g: input 32bit integer strong uniformity hypothesis: input ”1” passes → its correct for all integers weak uniformity hypothesis: tests for all integers 8 / 20

Verifying Test Hypotheses - HOL/TestGen Test Hypotheses Although we always use Test Hypotheses they are never explicit explicit Test Hypotheses explicit Test Hypotheses formalize common test practices and express the gab beetween test sucess and correctness [4] A test set should never be presented independently of its hypotheses [5] 9 / 20

Verifying Test Hypotheses - HOL/TestGen HOL/TestGen Workflow/System Four phases: writing the test specification TS (e.g: is sorted(SUT(list))) generation of test cases TC and test hyptheses for the TS generation of test data TD (variables replaced by ground instance) the test execution phase 10 / 20

Verifying Test Hypotheses - HOL/TestGen test case generation with explicit Test Hypotheses HOL/TestGen generates test cases toghether with the explicit hypotheses They give information about the quality of the test cases Test Hypotheses can be seen as a proof obligation for verification validation by test + proof of test hypotheses = ⇒ validation by proof 11 / 20

Verifying Test Hypotheses - HOL/TestGen test case generation with explicit Test Hypotheses uniformity hypothesis TS : if 0 ≤ x then SUT(x) else SUT(-x) test theorem 1 : 0 ≤ ? X 1 → SUT (? X 1) 2: THYP: ( ∃ x . 0 ≤ x → SUT ( x )) = ⇒ ( ∀ x . 0 ≤ x → SUT ( x )) 3: ? X 2 < 0 → SUT ( − ? X 2) 4: THYP: ( ∃ x . x < 0 → SUT ( − x )) = ⇒ ( ∀ x . x < 0 → SUT ( − x )) data generation phase will easily generate instances of the test cases e.g., SUT (3) and SUT ( − ( − 4)), (satisfying the constraints) 12 / 20

Verifying Test Hypotheses - HOL/TestGen test case generation with explicit Test Hypotheses regularity hypothesis TS : SUT ( list l ) test theorem 1 : SUT ([]) 2: SUT([?X1]) 3: SUT([?X2,?X3]) 4: THYP: ∀ x . 2 < | x | = ⇒ SUT ( x ) depth is user defined; in this example 2 uniformity hypotheses omitted 13 / 20

Verifying Test Hypotheses - HOL/TestGen DEMO DEMO TIME! example ”List” from HOL/TestGen 14 / 20

Verifying Test Hypotheses - HOL/TestGen testing Test Hypotheses increases quality of test cases weakens the hypothesis refeed test hypothesis into HOL/Testgen again generates more testcases and weakens the hypothesis. if we get rid of all hypotheses we would have to test the program with all possible inputs. 15 / 20

Verifying Test Hypotheses - HOL/TestGen proofing Test Hypotheses uniformity hypotheses proofed over real code all test cases passed + all hypotheses proofed → SUT verified uniformity hypothesis. ∃ x , xa . is sorted ( SUT ([ xa , x ])) = ⇒ ∀ x , xa . is sorted ( SUT ([ xa , x ])) SUT replaced by program sort, can be written as: � x , xa , x ′ , xa ′′ : is sorted ( sort ([ xa ′ , x ′ ])) = ⇒ is sorted ( sort ([ xa , x ])) The only way to proceed is by discarding the assumption � x , xa : is sorted ( sort ([ xa , x ])) ..... 16 / 20

Verifying Test Hypotheses - HOL/TestGen proofing Test Hypotheses uniformity hypotheses This shows that the test case is irrelevant for the proof of uniformity as the assumption (=test case) of the hypothesis had to be discharged test contributes to the proof only if it covers input space then the test is the verification 17 / 20

Verifying Test Hypotheses - HOL/TestGen proofing Test Hypotheses regularity hypotheses The uniformity hypotheses can be combined to ∀ l . | l | ≤ 3 = ⇒ is sorted ( sort ( l )) is sorted ( l ) → is sorted ( ins ( a , l )) regularity hypothesis by induction. anchor : � a , l . | l | = 3 = ⇒ is sorted ( ins ( a , sort ( l ))) step : � a , l . is sorted ( sort ( l )) = ⇒ is sorted ( ins ( a , sort ( l ))) 18 / 20

Verifying Test Hypotheses - HOL/TestGen Conclusion tests and (post-hoc) verifications can work together Test hypotheses can be seen as proof obligation that is proven in later stages of validation if needed Test hypotheses give information about the quality of tests testing test hypotheses can be used to approximate verification (covering input space) no hope for the desire to use tests to simplify proofs 19 / 20

Verifying Test Hypotheses - HOL/TestGen References Brucker, Br¨ ugger, Krieger, and Wolff. Hol-testgen 1.7.0 user guide. Laboratoire en Recherche en Infromatique (LRI),Universit e Paris-Sud , 2012. Brucker, Br¨ ugger, and Wolff. Verifying test-hypotheses - an experiment in test and proof. In Proceedings of the Fourth Workshop on Model Based Testing (MBT 2008) , 2008. Brucker and Wolff. Interactive testing with hol-testgen. Formal Approaches to Software Testing 5th International Workshop, FATES , 2005. M.-C. Gaudel G. Bernot and B. Marre. Software testing based on formal specifications: a theory and a tool. Software Engineering Journal, 6(6):387-405 , 1991. Marie-Claude Gaudel. Testing can be formal, too. Laboratoire en Recherche en Infromatique (LRI),Universit e Paris-Sud , 1995. 20 / 20