Contents � introduction & background � testing pre-orders Model-Based Testing � input/output & quiescence � ioco implementation relation Ed Brinksma � test generation � TorX � test case study University of Twente Dept. of Computer Science � real-time testing Formal Methods & Tools group Enschede The Netherlands ARTIST2 Summer School Nässlingen October 1, 2005 ARTIST2 Summer School 2 Contents Practical problems of testing � introduction & background Testing is: But also: � testing pre-orders important ad-hoc, manual, error-prone � � � input/output & quiescence much practiced hardly theory / research � � � ioco implementation relation 30% - 50% of project effort no attention in curricula � � expensive not cool : � test generation � � “if you’re a bad programmer time critical � � TorX you might be a tester” not constructive � � test case study (but sadistic?) e l ! ? b s i � real-time testing s o p Attitude is changing: s t s n e d m o h e t more awareness v e � o m r p l m a m I r more professional o � f h t w i October 1, 2005 ARTIST2 Summer School 3 October 1, 2005 ARTIST2 Summer School 4 Test Automation Types of Testing Level Traditional test automation Why not generate = tools to execute and manage test cases system test automatically?! integration specification TTCN TTCN test cases unit Accessibility pass white box black box robustness performance test implementation usability under test tool reliability fail functional behaviour Aspect October 1, 2005 ARTIST2 Summer School 5 October 1, 2005 ARTIST2 Summer School 6
Verification and Testing Testing with Formal Methods Verification : Testing : Testing with respect to a formal specification � formal manipulation experimentation � � Precise, formal definition of correctness : � prove properties show error � � good and unambiguous basis for testing performed on model concrete system � � Formal validation of tests � formal concrete Algorithmic derivation of tests : � world world tools for automatic test generation Allows to define measures expressing coverage � Verification is only as good as Testing can only show the Verification is only as good as and quality of testing Testing can only show the the validity of the model on presence of errors, not their the validity of the model on presence of errors, not their which it is based absence which it is based absence October 1, 2005 ARTIST2 Summer School 7 October 1, 2005 ARTIST2 Summer School 8 Challenges of Testing Theory Formal Testing i imp s � Infinity of testing: specification S � too many possible input combinations -- infinite breadth ⇑ ⇓ sound ⇔ exhaustive � too many possible input sequences -- infinite depth test i passes T s � too many invalid and unexpected inputs correctness generation criterion implementation � Exhaustive testing never possible: relation � when to stop testing ? imp test suite T S � how to invent effective and efficient test cases with high probability of detecting errors ? implementation i � Optimization problem of testing yield and invested effort test pass / fail execution � usually stop when time is over ...... October 1, 2005 ARTIST2 Summer School 9 October 1, 2005 ARTIST2 Summer School 10 Contents Formal Testing : Conformance � introduction & background � testing pre-orders s ∈ SPECS Specification specification S IUT Implementation under Test � input/output & quiescence � ioco implementation relation IUT conforms-to s IUT is concrete, physical object � test generation correctness criterion � TorX Model the physical world � test case study But IUT is black box ! ? implementation � real-time testing IUT Assume that model i IUT exists October 1, 2005 ARTIST2 Summer School 11 October 1, 2005 ARTIST2 Summer School 12
Testing Preorders Classical Testing Preorder on Transition Systems ≤ te implementation specification ≤ implementation specification i s i s environment environment environment environment e e e e For all environments e i ≤ te s ∀ e ∈ E . obs ( e, i ) ⇔ ⊆ obs (e, s ) all observations of an implementation i in e ∀ e ∈ Env . obs ( e, i ) i ≤ s ⇔ ⊆ obs (e, s ) should be explained by ↓ ↓ ↓ ↓ ↓ observations of the specification s in e. LTS(L) Deadlocks(e||s) ? ? ? October 1, 2005 ARTIST2 Summer School 13 October 1, 2005 ARTIST2 Summer School 14 Classical Testing Preorder Quirky Coffee Machine [Langerak] ≤ te implementation specification Can we distinguish between these machines? i s coin coin coin coin environment environment coffee coffee e e tea tea ≈ te bang bang bang bang tea tea coffee coffee i ≤ te s i ≤ te s ∀ e ∈ LTS(L) . ∀ σ ∈ L* . ∀ e ∈ LTS(L) . ∀ σ ∈ L* . ⇔ ⇔ e ||i deadlocks after σ ⇒ e ||s deadlocks after σ { σ | e || i deadlocks after σ } ⊆ They are { σ | e||s deadlocks after σ } ⇔ FP (i) ⊆ FP (s) testing equivalent! FP (p) = { 〈 σ , A 〉 | p afer σ refuses A } October 1, 2005 ARTIST2 Summer School 15 October 1, 2005 ARTIST2 Summer School 16 Quirky Coffee Machine Refusal Preorder Revisited ≤ rf implementation specification coin coin coin coin i s ≈ te coffee coffee tea tea Deadlocks δ (e||i) = bang bang bang bang { σ∈ (L ∪ { δ })* | ≈ rf environment environment coffee tea tea coffee e e ||i deadlocks after σ } e coin i ≤ rf s ∀ e ∈ E . obs ( e, i ) ⇔ ⊆ obs (e, s ) δ only enabled coffee δ ↓ ↓ if coffee is not e observes with δ bang LTS ( L ∪ { δ }) Deadlocks δ (e||i) deadlock on all coffee alternative actions tester October 1, 2005 ARTIST2 Summer School 17 October 1, 2005 ARTIST2 Summer School 18
Contents I/O Transition Systems � introduction & background � testing pre-orders testing actions are usually directed, i.e. there are inputs and � outputs � input/output & quiescence L=L in ∪ L out with L in ∩ L out = ∅ � ioco implementation relation systems can always accept all inputs (input enabledness) � test generation � a � TorX for all states s, for all a ∈ L in s ⇒ � test case study testers are I/O systems � � real-time testing � output (stimulus) is input for the SUT � input (response) is output of the SUT October 1, 2005 ARTIST2 Summer School 19 October 1, 2005 ARTIST2 Summer School 20 Quiescence Input-Output QCM states must coin! Because of input enabledness S||T deadlocks iff T � be saturated quiescent produces no stimuli and S no responses. This is known as with input states coin? coin? coffee! coffee? quiescence loops for tea? coffee? input Observing quiescence leads to two implementation relations � δ tea? coffee? enabledness bang? bang? for I/O systems I and S: 1. I ≤ iote S iff for all I/O testers T: coffee! tea ! bang! Deadlocks(I||T) ⊆ Deadlocks(S||T) � coffee? tea? (quiescence) ≈ iote coffee ! 2. I ≤ iorf S iff for all I/O testers T: coffee! tea ! ≈ iorf coffee? Deadlocks δ (I||T) ⊆ Deadlocks δ (S||T) � (repetitive quiescence) October 1, 2005 ARTIST2 Summer School 21 October 1, 2005 ARTIST2 Summer School 22 Contents Implementation Relation ioco � introduction & background δ � testing pre-orders By adding a transition p → p to every quiescent state of a system we treat quiescence as an observable (synchronizable) action: � input/output & quiescence � ioco implementation relation i ≤ iorf s ⇔ ∀ I/O tests T : Deadlocks δ (i|| T ) ⊆ Deadlocks δ (s|| T ) � test generation ⇔ ∀σ ∈ ( L ∪ { δ } )*: out ( i afte r σ ) ⊆ out ( s after σ ) � TorX � test case study To allow under-specification we restrict the set of traces: � real-time testing i ioco s ⇔ ∀σ ∈ Traces δ ( s ) : out ( i afte r σ ) ⊆ out ( s after σ ) October 1, 2005 ARTIST2 Summer School 23 October 1, 2005 ARTIST2 Summer School 24
Recommend
More recommend
