Software Testing for Critical Systems Julien Fayolle and Sandrine-Dominique Gouraud 1 1 G´ enie logiciel, LRI, Universit´ e d’Orsay. { fayolle, gouraud } @lri.fr Web pages: www.lri.fr/ ∼ fayolle and www.lri.fr/ ∼ gouraud University of York, CS seminar. Fayolle & Gouraud Software Testing for Critical Systems 1
Overview of Testing How does one test a software? ➪ Testing all entries is not possible. ➪ We don’t prove the software right. ➘ Create a test set . Functional testing: specifications, black box Structural testing: structure and code, glass box Statistical testing: draw at random from the input domain ➘ Run the program on the test set. ➘ Compare the outputs to the expected outputs (oracle). Fayolle & Gouraud Software Testing for Critical Systems 2
Statistical Testing Software for critical systems (planes’ motors, nuclear plants’ safety, missiles, . . . ); Intensive statistical testing; Provide quantitative figures on the test set; Provide adequacy figures on the test methods; Drawback: coverage of rare cases. factorial(integer n) if(n!=0) fact:=1; for i from n to 2 fact:=fact*i; else fact:=0; Fayolle & Gouraud Software Testing for Critical Systems 3
Statistical Structural Testing First answer: [Th´ evenod-Fosse & Waeselynck’91, LAAS] ➠ Combining statistical and structural testing. ➠ Compute a distribution on the input domain and draw tests inputs accordingly. ➠ How representative is the test set? Fayolle & Gouraud Software Testing for Critical Systems 4
Test quality 1 6 j a c g e h 0 3 4 7 b f k 2 5 d Test quality q n = 1 − (1 − q ) n , where n is the number of tests and q the minimal probability to reach one element. Fayolle & Gouraud Software Testing for Critical Systems 5
AuGuSTe Drawback: the input distribution has to be computed by hand. AuGuSTe . ➔ Automated Generation of Statistical Tests , [Gouraud et alii ’04] First programmed for C code. Coded in Objective Caml; uses GAT E L, MuPAD, Eclipse, Prolog, C, and Java. Statistical structural testing. Drawing uniformly at random paths of a given length in the control graph. AuGuSTe web site: http://www.lri.fr/ ∼ fayolle/Auguste.html Fayolle & Gouraud Software Testing for Critical Systems 6
Structural coverage criteria Given a control flow graph All paths All i -paths Branch coverage: all edges Block coverage: all instructions Predicate coverages Test quality for statistical structural tests q n minimal probability to reach one element with a test set of size n . q n = 1 − (1 − q ) n . Fayolle & Gouraud Software Testing for Critical Systems 7
Random generation of paths Goal: optimize the test quality i.e. maximize the smallest probability to reach an element. Combinatorial description of the control graph Uniform drawing of path (criteria consideration) 1 6 j a c g e 0 3 4 7 f h b k 2 5 d Atoms are edges G = ac ( egj + fG 1 ) + bdG 1 , and G 1 = hj + k . Fayolle & Gouraud Software Testing for Critical Systems 8
How does AuGuSTe work? Inputs Program Coverage criteria Number of tests (test quality) Maximal length of the paths (elementary path) 1. Analysis: building the combinatorial structure of the control graph 2. Drawing paths uniformly at random in the graph 3. Constraint resolution (GAT E L) Output Test set. Fayolle & Gouraud Software Testing for Critical Systems 9
Testing methods comparison using mutants Creating mutants : seed faults in the original program. A large repertoire of mutation operators: changing ≥ in ≤ , > or < ; changing 0 in 2; performance criteria: mutation score (kill count). stability of the fault detection ability ➳ average. Softwares for creating mutants Mothra [Offutt, DeMillo et alii ’88]; SESAME [Crouzet, Th´ evenod-Fosse & Waeselynck’98]; µ Java [Offutt, Ma & Kwon’03]. Fayolle & Gouraud Software Testing for Critical Systems 10
Experimental results Function from industrial software (77 lines) 605 mutants (for some mutations). Test quality .9999 5 runs of test sets of size 850. min avg max Uniform testing 0.8950 NA 0.9150 Structural statistical testing 0.9898 0.9901 0.9915 AuGuSTe 0.9854 0.9854 0.9854 Table: Experimental mutation scores over 5 runs Fayolle & Gouraud Software Testing for Critical Systems 11
GAT E L GAT E L ➔ Test Suites Generation from Lustre Descriptions [Marre et alii , CEA and LRI] Uses Lustre descriptions. Lustre code or Lustre spec. (test) Uses constraint solving techniques, non-deterministic heuristics. Structural or functional software testing. GAT E L web site : http://www-list.cea.fr/labos/gb/LSL/test/gatel/index.html Fayolle & Gouraud Software Testing for Critical Systems 12
Lustre code node Alarm(stream0, s up, s down: real ) returns (alarm s: bool ); var upstream, downstream: bool let upstream = stream0 > s up; downstream = stream0 < s down ; alarm s = Automaton(upstream, upstream, downstream); tel; node Automaton(init, upstream, downstream: bool ) returns (state: bool ); let state = init - > if upstream and not pre (state) then true else if downstream and pre (state) then false else pre (state); let ; Fayolle & Gouraud Software Testing for Critical Systems 13
Lustre Declarative language for specification and/or programming. Reactive synchronous systems. Data-flow. The specification allows the automatic generation of code. Fayolle & Gouraud Software Testing for Critical Systems 14
Perspectives ✭ Extend the idea of uniform drawing to the synchronous data-flow languages. ✭ Application to Lustre ✭ Enrich GAT E L with random generation of combinatorial structures. ✭ Provide precise bounds on the fault detection ability of AuGuSTe-like methods for imperative and declarative languages. Fayolle & Gouraud Software Testing for Critical Systems 15
Recommend
More recommend
