Revisiting coverage criteria for Scade models Jean-Louis Cola co 7 - PowerPoint PPT Presentation

Revisiting coverage criteria for Scade models Jean-Louis Cola¸ co 7 December 2016

Context ◮ Code coverage is a measure that characterises how much a given test suite exercises a code , ◮ lots of criteria exist, avioncs standard (DO-178) requires MC/DC for the most critical application, ◮ in DO-178C (2011), suplement DO-331 about Model Based Design now requires model coverage. ◮ SCADE proposes model coverage for about 10 years: ◮ was based on ad’hoc criteria defined by the user per operator, ◮ recent solution is inspired by work of Parissis et al. A. Lakehal and I. Parissis, Structural coverage criteria for LUSTRE/SCADE programs, in Software Testing, Verification and Reliablity , Wiley Interscience, 2009 J-L. Camus, C. Haudebourg and M. Schlickling Data Flow Model Coverage Analysis: Principles and Practice in Embedded Real Time Software and Systems , 2016 c 2 � ANSYS, Inc.

Why revisiting? ◮ current solution is based on Paths in the dataflow: quite complex objects; ◮ to study the relationship between model coverage and generated code coverage: paths are not well suited; ◮ to overcome some limitation of current implementation. c 3 � ANSYS, Inc.

Why revisiting? ◮ current solution is based on Paths in the dataflow: quite complex objects; ◮ to study the relationship between model coverage and generated code coverage: paths are not well suited; ◮ to overcome some limitation of current implementation. The idea we had for the rework was actually nicely presented in: M. Whalen, G. Gay, Y. Dongjiang, M. P.E. Heimdahl and M. Staats Observable modified condition/decision coverage in Proceedings of the 35th International Conference on Software Engineering , 2013 c 3 � ANSYS, Inc.

Why revisiting? ◮ current solution is based on Paths in the dataflow: quite complex objects; ◮ to study the relationship between model coverage and generated code coverage: paths are not well suited; ◮ to overcome some limitation of current implementation. The idea we had for the rework was actually nicely presented in: M. Whalen, G. Gay, Y. Dongjiang, M. P.E. Heimdahl and M. Staats Observable modified condition/decision coverage in Proceedings of the 35th International Conference on Software Engineering , 2013 present work continues and extends it to full Scade 6 language. c 3 � ANSYS, Inc.

Agenda Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion c 4 � ANSYS, Inc.

Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion c 5 Intuition � ANSYS, Inc.

Glossary flow or stream : infinite sequence of values. model : a Scade program and a root node . monitor : any construction that allows to observe a flow out of the model: (root node) outputs, probes, . . . outcome (of a test) values taken by all the monitors of the model when running a test. source designates any construction that introduces flow that that does not result from the combination of other flows. (root node) inputs, sensors, literal values, reference to constants. c 6 Intuition � ANSYS, Inc.

The intuition ◮ Covering a stream occurrence s requires exhibiting a test that shows its ability to influence a monitor (red bubles); ◮ Covering a model requires covering all its streams occurrences . c 7 Intuition � ANSYS, Inc.

Criterion 1: Influence A test T shows the influence of stream x of a model M if: ◮ T is such that x is in situation to influence an output of M ◮ i.e. T is such that modifying stream x in the execution of the test changes the outcome. A test suite T S covers a model M if for all stream x of M , T S contains a test T that covers stream x . c 8 Intuition � ANSYS, Inc.

Criterion 2: OMC/DC A pair of tests ( T 1 , T 2 ) satisfies OMC/DC criterion for a Boolean stream b of a model M if T 1 and T 2 are such that: ◮ b takes different values in each test case and ◮ toggling b in both test cases changes the outcome. A test suite T S covers a model M in the sense of OMC/DC if for all Boolean stream b of M , T S contains two tests T 1 and T 2 such that satisfy the condition above. c 9 Intuition � ANSYS, Inc.

Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion c 10 Ideal definition of coverage � ANSYS, Inc.

Notations ◮ D n represent the set of stream prefix of size smaller or equal to n . ◮ If x is a stream prefix, | x | represents its size. ◮ If x is a stream prefix, ( x ) i where i ≤ | x | represents i th value. ◮ Let M be a Scade model and n in its number of inputs. ◮ A test case T of length n cycle is a tuple of n in components of D n . ◮ M ( T ) represents the execution of test case T ; the outcome of this execution is itself a tuple of values in D n (one per monitor). ◮ If v is a stream prefix of a Boolean stream, ¬ i ( v ) represents the prefix with same length built from v by negating its i th value. ◮ A stream occurrence is represented as ⌊ e ⌋ k where k is an integer and e is a stream expression. c 11 Ideal definition of coverage � ANSYS, Inc.

Occurrences identification Defined by function Streams ( . ): Streams ( x1, ..., xn = e ; ) def = Streams ( e ) def · · · = · · · def � � Streams ( x ) = ⌊ x ⌋ k def � � Streams ( 1 ) = ⌊ 1 ⌋ k def Streams ( ’s; ) = {⌊ ’s ⌋ k } def Streams ( last ’s; ) = {⌊ last ’s ⌋ k } = {⌊ op ( e 1 , . . . , e n ) ⌋ k } � Streams ( e 1 ) def � Streams ( op ( e 1 , . . . , e n )) . . . def · · · = · · · c 12 Ideal definition of coverage � ANSYS, Inc.

Occurrences identification example Streams ( o = x*x + pre (2*x) + 1; ) =   ⌊ x ⌋ 1 , ⌊ x ⌋ 2 , ⌊ x ⌋ 3 , ⌊ 2 ⌋ 4 , ⌊ 1 ⌋ 5 ,           � � ⌊⌊ x ⌋ 1 ∗ ⌊ x ⌋ 2 ⌋ 6 , ⌊⌊ 2 ⌋ 4 ∗ ⌊ x ⌋ 3 ⌋ 7 , ⌊ pre ( ⌊ 2 ⌋ 4 ∗ ⌊ x ⌋ 3 ) ⌋ 7   8 ,       � � � � ⌊⌊ x ⌋ 1 ∗ ⌊ x ⌋ 2 ⌋ 6 + ( pre ⌊⌊ 2 ⌋ 4 ∗ ⌊ x ⌋ 3 ⌋ 7 9 ,  8           �� �    � � � ⌊⌊ x ⌋ 1 ∗ ⌊ x ⌋ 2 ⌋ 6 + ( pre ⌊⌊ 2 ⌋ 4 ∗ ⌊ x ⌋ 3 ⌋ 7 9 + ⌊ 1 ⌋ 5     8 10 c 13 Ideal definition of coverage � ANSYS, Inc.

Stream occurrence mutation Let M be a model where: ◮ ⌊ e ⌋ k one of its stream occurrences: ⌊ e ⌋ k ∈ Streams ( M ), ◮ v is a finite stream prefixe: v ∈ D n , ◮ e and v are of same type, ◮ e ′ is a stream expression with same clock as e : e e 0 · · · e n e n +1 e n +2 · · · · · · v v 0 v n e ′ v 0 · · · v n e n +1 e n +2 · · · M ( v ◮ ⌊ e ⌋ k ) represents the model obtained by substituting ⌊ e ⌋ k in M by a e ′ ; we called it a mutant of M for the occurrence ⌊ e ⌋ k . c 14 Ideal definition of coverage � ANSYS, Inc.

Influence ideal definition Coverage of stream x by T : def ∃ n > 0 . ∃ v ∈ D n . M ( T ) � = M ( v ◮ x ) ( T ) Influence ( T , x , M ) = Coverage of model M by a test suite T S : ∀ x ∈ Streams ( M ) . ∃ T ∈ T S . Influence ( T , x , M ) c 15 Ideal definition of coverage � ANSYS, Inc.

OMC/DC Ideal definition Coverage of stream x by ( T 1 , T 2 ): def Omcdc ( T 1 , T 2 , b , M ) = ( b T 1 ) i � = ( b T 2 ) j   M ( T 1 ) � = M ( ¬ i ( b T 1 ) ◮ b ) ( T 1 ) � ∃ ( i , j ) ∈ N × N .     M ( T 2 ) � = M ( ¬ j ( b T 2 ) ◮ b ) ( T 2 ) � Coverage of model M by a test suite T S : ∀ b ∈ Streams ( M ) . � � ∃ ( T 1 , T 2 ) ∈ T S × T S . ( b : bool) ⇒ Omcdc ( T 1 , T 2 , b , M ) c 16 Ideal definition of coverage � ANSYS, Inc.

Limit of the ideal definition Not really implementable: ◮ based on the exitence of mutants without giving a way to build them (it is a guess); ◮ requires both executions on original model and on the mutant; ◮ needs one mutant per stream occurrence. c 17 Ideal definition of coverage � ANSYS, Inc.

Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion c 18 Scade tagged semantics � ANSYS, Inc.

Tagged semantics Tagged semantics: ◮ based on tagged values; ◮ defines tag propagation rules. ◮ provides primitives for tag introduction; c 19 Scade tagged semantics � ANSYS, Inc.

Tagged values The values used in a tagged Scade model M # are in V # n , m defined by: ( bool � numeric � � def V # � = declared enum values ) × P ( Tags ) 0 , m def V # V # = n , m n +1 , m � � [ v # 1 , . . . , v # � 1 ≤ i ≤ p ≤ m , v # ∈ V # � � p ] × P ( Tags ) n , m i � � { l 1 : v # 1 , . . . , l p : v # � 1 ≤ i ≤ p ≤ m , v # ∈ V # � � × P ( Tags ) p } n , m i where Tags is a finite set of tags c 20 Scade tagged semantics � ANSYS, Inc.

Tag propagation of combinatorial operators For most operators input tags propagate to the outputs: op # (( v 1 , τ 1 ) , . . . , ( v n , τ n )) = ( op ( v 1 , . . . , v n ) , � i ∈ [1 .. n ] τ i ) c 21 Scade tagged semantics � ANSYS, Inc.