Combining Verification and Conformance Testing for Validating Reactive Systems Vlad Rusu, Thierry Jéron, and Hervé Marchand First.Last@irisa.fr IRISA/INRIA Rennes, project Vertecs http://www.irisa.fr/vertecs Combining verification and conformance testing - ETR’05 – p.1/38
Verification Properties: P S | = P Specification: S Combining verification and conformance testing - ETR’05 – p.2/38
Conformance testing Specification: S I ioco S Implementation: I Combining verification and conformance testing - ETR’05 – p.3/38
Model-based development Properties: P Verification Specification: S Testing Implementation: I Combining verification and conformance testing - ETR’05 – p.4/38
Model-based development Properties: P Verification Verification? Specification: S Testing Implementation: I Combining verification and conformance testing - ETR’05 – p.5/38
Model-based development Properties: P Verification Testing Specification: S Testing Implementation: I Combining verification and conformance testing - ETR’05 – p.6/38
Outline verification conformance testing combining the two: consistency? test generation. Combining verification and conformance testing - ETR’05 – p.7/38
Verification Properties: P S | = P Specification: S properties: safety , liveness . . . specification: SDL, Statecharts, extended automata . . . Combining verification and conformance testing - ETR’05 – p.8/38
Verification techniques model checking: automatic, finite-state, exact theorem proving: interactive, infinite-state, exact abstract interpretation : automatic, infinite-state, approximated (conservative) Combining verification and conformance testing - ETR’05 – p.9/38
Sample specification START ?( p ) x := p m = x ∧ x > 0 MSG !( m ) x := x − 1 x = 0 STOP ! Combining verification and conformance testing - ETR’05 – p.10/38
Property: after START ?( p ) with p > 0 , no STOP ! without MSG !( m ) in between ∗ p > 0 MSG !( m ) START ?( p ) START ?( p ) x := p m = x ∧ x > 0 MSG !( m ) ∗ x := x − 1 x = 0 STOP ! STOP ! Violate Combining verification and conformance testing - ETR’05 – p.11/38
Verifying Property on Specification check reachability of Violate in synchronous product (Spec × observer for property) undecidable (abstract interpretation: conservative) here, property is proved, but not proved if p ≥ 0 only. Combining verification and conformance testing - ETR’05 – p.12/38
Outline verification conformance testing conformance relation test generation and execution combining the two: consistency? test generation. Combining verification and conformance testing - ETR’05 – p.13/38
Conformance testing Specification: S I ioco S Implementation: I ioco ≅ after all traces of S , outputs of I ⊆ outputs of S . Combining verification and conformance testing - ETR’05 – p.14/38
Conformance and non-conformance ¬ ( I 2 ioco S ) S I 1 ioco S I 3 ioco S ST ART ?( p ) x := p ST ART ?(0) ST ART ?(0) ST ART ?(0) m = x ∧ x > 0 MSG !( m ) x := x − 1 ST ART ?(0) ST OP ! MSG !(0) x = 0 ST OP ! MSG !(0) Combining verification and conformance testing - ETR’05 – p.15/38
Conformance? S I START ?( p ) x := p START ?(0) m = x ∧ x > 0 Quiet... MSG !( m ) x := x − 1 x = 0 STOP ! Combining verification and conformance testing - ETR’05 – p.16/38
Blocking and suspension blocking δ ! : deadlock, or waiting for input δ ! observed on black-box I (using timers) computed on S : suspension operation δ ( · ) Traces ( δ ( S )) = Traces ( S ) + δ ! when blocked. Combining verification and conformance testing - ETR’05 – p.17/38
Non-Conformance! δ ( S ) δ ! δ ( I ) START ?( p ) x := p START ?(0) x < 0 m = x ∧ x > 0 δ ! MSG !( m ) δ ! x := x − 1 x = 0 STOP ! δ ! Combining verification and conformance testing - ETR’05 – p.18/38
Conformance relation I ioco S � Traces ( δ ( I )) ∩ Traces ( δ ( S )) · (Λ ! S ∪ { δ } ) ⊆ Traces ( δ ( S )) . “After all traces of δ ( S ) , outputs of δ ( I ) ⊆ outputs of δ ( S ) ” Combining verification and conformance testing - ETR’05 – p.19/38
Test generation & execution selects Test Purposes Specification S ioco Test Generation Implementation I Test Cases Test Execution Verdicts (e.g., Fail, Pass, Inconclusive) Combining verification and conformance testing - ETR’05 – p.20/38
Test Purpose and Test Case Test Purpose Test Case START ?( p ) START !(0) MSG ?( m ) ∗ Fail δ ? STOP ! STOP ? Pass Combining verification and conformance testing - ETR’05 – p.21/38
Outline verification conformance testing combining the two: consistency guide implementation towards property violation deal with “imperfect” verification test generation. Combining verification and conformance testing - ETR’05 – p.22/38
Using P in test generation | = Safety Properties P Specification S Verification/ ioco Test Generation Implementation I Test Cases Test Execution Verdicts (e.g., Fail, Violate, Inconclusive) Combining verification and conformance testing - ETR’05 – p.23/38
Problem: S | = P undecidable try to prove S | = P using abstract interpretation even if S | = P could not be proved, generate test case that correctly detects ¬ I ioco S , I �| = P , or S �| = P use symbolic techniques for test generation as well. Combining verification and conformance testing - ETR’05 – p.24/38
Outline verification conformance testing combining the two: consistency? test generation: first define a canonical tester for S and ioco then compute product with observer for property finally, analyse product, eliminate irrelevant parts. Combining verification and conformance testing - ETR’05 – p.25/38
Canonical tester: determinisation Determinisation: det ( · ) det ( S ) is deterministic Traces ( det ( S )) = Traces ( S ) . Combining verification and conformance testing - ETR’05 – p.26/38
Canonical tester: output-completion Output-completion: Σ ! ( · ) add a new location: Fail & transitions to it on each missing output notation: canon ( S ) = Σ ! ( det ( δ ( S ))) Combining verification and conformance testing - ETR’05 – p.27/38
canon ( S ) δ ! *! START ?( p ) x := p x < 0 δ ! *! m = x ∧ x > 0 Fail MSG !( m ) x := x − 1 x = 0 STOP ! δ ! *! canon ( S ) = canonical tester [Brinksma] for S and ioco . ioco -conformance to S is a safety property! Combining verification and conformance testing - ETR’05 – p.28/38
Test generation: product with observer For observer ( ω, Violate ω ) let test ( S , ω ) = ω || canon ( S ) . Three possible sets of violating locations: ViolateFail = Violate ω × { Fail } Fail = Violate ω × { Fail } Violate = Violate ω × { Fail } Combining verification and conformance testing - ETR’05 – p.29/38
The ViolateFail verdict δ ( I ) �| = ( test ( S , ω ) , ViolateFail ) ⇒ ¬ ( I ioco S ) ∧ δ ( I ) �| = ( ω, Violate ω ) Implementation violates both property and conformance Combining verification and conformance testing - ETR’05 – p.30/38
The Fail verdict δ ( I ) �| = ( test ( S , ω ) , Fail ) = ⇒ ¬ ( I ioco S ) Implementation violate conformance only Combining verification and conformance testing - ETR’05 – p.31/38
The Violate verdict δ ( I ) �| = ( test ( S , ω ) , Violate ) ⇒ δ ( S ) , δ ( I ) �| = ( ω, Violate ω ) Specification & implementation violate property Combining verification and conformance testing - ETR’05 – p.32/38
Specification and property δ ! ∗ p ≥ 0 START ?( p ) MSG !( m ) START ?( p ) x := p x < 0 δ ! m = x ∧ x > 0 MSG !( m ) ∗ x := x − 1 x = 0 STOP ! STOP ! δ ! Violate Combining verification and conformance testing - ETR’05 – p.33/38
Resulting test case ?* Fail ?* ?* δ ? ?* p ≥ 0 x < 0 START !( p ) δ ? x := p MSG ?( m ) m = x ∧ x > 0 x < 0 m = x ∧ x > 0 MSG ?( m ) δ ? x := x − 1 x := x − 1 x � = 0 x = 0 x = 0 STOP ? STOP ? STOP ? Violate Violate Fail Combining verification and conformance testing - ETR’05 – p.34/38
Simplifying a test case Eliminate states from which property cannot be violated (from which Violate and ViolateFail are not reachable). abstract interpretation strikes again! coreachability analysis to Violate and ViolateFail ❀ constraints in each transition strengthen guards of transitions with new constraints only non-coreachable states are eliminated (but some may remain). Combining verification and conformance testing - ETR’05 – p.35/38
Simplified test case *? Fail p ≥ 0 *? START !( p ) x := 0 MSG ?( m ) Inconc x > 0 x = 0 STOP ? STOP ? Violate Violate Fail Combining verification and conformance testing - ETR’05 – p.36/38
Conclusion combination of verification and conformance testing automatic, symbolic methods testing provides correct verdicts, regardless of success in verification step test case “guides” implementation towards violating property. Only one among many approaches! Combining verification and conformance testing - ETR’05 – p.37/38
Questions? Combining verification and conformance testing - ETR’05 – p.38/38
Recommend
More recommend
