Model-Based Testing There is Nothing More Practical than a Good - - PowerPoint PPT Presentation
Model-Based Testing There is Nothing More Practical than a Good Theory model Jan Tretmans SUT ESI Embedded Systems Innovation by TNO Radboud University Nijmegen Hgskolan i Halmstad pass fail jan.tretmans@tno.nl Jan Tretmans Radboud
Jan Tretmans
ESI – Embedded Systems Innovation by TNO Radboud University Nijmegen Högskolan i Halmstad jan.tretmans@tno.nl
SUT
pass fail
model
ESI Embedded Systems Innovation (TNO) Eindhoven The Netherlands Radboud University Nijmegen The Netherlands
ESI RU UT
3
1. Introduction
– Motivation – Basics: what is it – Context: Model-Based Verification, Validation, Code Generation, . . .
2. Theory
MBT with Labelled Transition systems
– models – implementation relation ioco – testing for ioco 3. Tools
– TorXakis - a tool voor ioco testing – Representation of models : Language
4. Applications
– dropbox
4
5
connectivity multi disciplinarity change variability evolvability complexity uncertainty heterogeneous components
6
model composition abstraction under
specification
uncertainty
nondeterminism
concurrency
parallelism
state + complex data
statistical usage profiles
partial
specification
multiple paradigms
integration
test selection
Model Based Testing
7
Checking or measuring some quality characteristics
by performing experiments in a controlled way w.r.t. a specification
specification
System Under Test
specification-based, active, black-box testing of functionality
8
System Under Test pass fail 1. Manual testing
9
pass fail test execution
TTCN TTCN
test cases 1. Manual testing 2. Scripted testing
10
pass fail test execution 1. Manual testing 2. Scripted testing 3. Keyword-driven testing
high-level test notation
11
test scripts
system model
TTCN TTCN
Test cases pass fail model-based test generation test execution 1. Manual testing 2. Scripted testing 3. Keyword-driven testing 4. Model-based testing
12
?coin ?button !alarm ?button !coffee 13
MBT: next step in test automation
+ test execution + result analysis
more variation in test flow and in test data
unambiguous analysis of test results
improved regression testing
model coverage customer profile coverage
detecting more bugs faster and cheaper
SUT
pass fail model-based test generation
test execution
model
16
17
making a model reveals errors
go step-by-step through the model
go through all states of the model
prove theorems about the model
executable code from the model
test an implementation for compliance
generate a model from observation
informal world real world
informal requirements
18
formal world
19
A model is more (less) than code generation:
20
21
realization virtualization design models abstract (test) models
22
system model
TTCN TTCN
Test cases pass fail
test execution model-based test generation SUT conforms to model
23
tester environment mechanical physical
system
software
24
tester environment model mechanical physical model
system
software
software model
25
26
27
states transitions T S (L {}) S initial state s0 S Labelled Transition System: S, LI, LU, T, s0 input labels
labels
? = input ! = output ?coin ?button !alarm ?button !coffee
28
29
LTS model
behaving as input-enabled LTS TTCN TTCN
Test cases pass fail LTS test execution ioco test generation
input/output conformance
ioco
set of LTS tests
SUT passes tests SUT ioco model
sound exhaustive
30
p
p = !x LU {} . p
!x
!x
, pP } { | p
p, pP } Straces ( s ) = { ( L {} )* | s
} p after = { p’ | p
p’ }
i ioco s =def Straces (s) : out (i after ) out (s after ) s is a Labelled Transition System i is (assumed to be) an input-enabled LTS
31
!coffee
?dime ?quart ?dime ?quart ?dime ?quart ?dime
!choc
?quart !tea
!coffee
?dime !tea
specification model
?dime
!coffee
?dime
!choc
?dime !tea
non-determinism uncertainty under-specification
32
? x (x >= 0) ! y ( | y x y – x| < ε ) specification model
! x
? x (x < 0) ? x (x >= 0) SUT models ? x
rather than construction
! -x
? x (x < 0) ? x (x >= 0) ? x !error
33
34
– ‘quiescence’ / ‘time-out’ label – tree-structured – finite, deterministic – final states pass and fail – from each state pass, fail :
!dub !kwart ?tea ?coffee ?tea
!dub
fail fail
test case = labelled transition system
fail fail
?coffee ?tea
fail pass
?coffee ?tea
fail fail
?coffee ?tea ?coffee
pass fail pass
35
Algorithm to generate a test case t(S) from a transition system state set S, with S ( initially S = s0 after ). Apply the following steps recursively, non-deterministically:
1 end test case
pass
2 supply input !a !a t ( S after ?a )
allowed outputs (or ): !x out (S) forbidden outputs (or ): !y out (S)
3
fail
t ( S after !x )
fail
allowed outputs forbidden outputs
?y
?x fail
t ( S after !x )
fail
allowed outputs forbidden outputs
?y ?x
36
i ioco s i fails t !coffee
?dime !tea
specification model s
?coffee !dime ?tea ?choc
pass fail pass fail
generated test case t !choc
?dime !tea
implementation i
i ioco s =def Straces (s) :
37
SUT
behaving as input-enabled LTS
i ioco s test tool
gen : LTS → (TTS)
t SUT
SUT passes gen(s) SUT comforms to s
sound exhaustive
Prove soundness and exhaustiveness:
mIOTS .
( tgen(s) . m passes t )
m ioco s
Test assumption :
SUTIMP . mSUT IOTS . tTESTS .
SUT passes t mSUT passes t
pass fail
38
39
40
TorXakis
41
under
specification
partial
specification
uncertainty
nondeterminism
multiple paradigms
integration
test selection model composition state + complex data abstraction concurrency parallelism
statistical usage profiles SUT
TorXakis
model
a?n a?k x!n+1 x!p [[p<n]] b?m y!`no` a?n y!`yes`
model
42
modelled as state-transition system
black-box system view
43
44
{C10,Coffee,Tea} , { (S0,C10,S1), ( S1, Coffee, S2 ), (S1,Tea,S3) } , S0
S ::= C10 >-> ( Coffee ## Tea )
Coffee C10 Tea S1 S2 S3 S0
S
45
A B A B
A >-> B A ## B
STOP
A >-> B | | | C >-> D
D A C B D C A B D C A B
A A A A A B B B B
p where p ::= A >-> p q where q ::= A >-> ( B | | | q )
46
47
in ? n :: int [[ n ≠ 0 ]]
[[ 0 < m < n ]]
[[ 0 < m < -n ]]
semantics
in-2 in-1 in3 in2 in1 in-3
in4 in-4
basic behaviour = transition system
48
complex behaviour = process-algebraic composition
A ?n A ?k X !n+1 X !p [[p<n]] B ?m Y !`no` A ?n Y !`yes`
49
Models
50
Tool
Under the hood
for symbolic transition systems
But ....
Current Research
composition https://github.com/TorXakis
Applications
systems companies
51
Systems
Testing
keyword-driven test automation
keyword-driven test automation SUT
with simulated hardware/physics Models
53
Mysteries of DropBox: Property-Based Testing
In IEEE ICST, pp. 135–145. 1. Dropbox 2. Testing approach 3. Model 4. Test runs
54
55
56
Dropbox Client Dropbox Client 1
Dropbox Client 2 Read0 Write0 Read1 Write1 Read2 Write2
connectivity multi disciplinarity change variability evolvability complexity uncertainty heterogeneous components
57
model composition abstraction under
specification
uncertainty
nondeterminism
concurrency
parallelism
state + complex data
statistical usage profiles
partial
specification
multiple paradigms
integration
test selection
Model Based Testing
58
localVal0 clean0 fresh0 localVal1 clean1 fresh1
serverVal conflicts
localVal2 clean2 fresh2
𝐽𝑜 ? 𝑆𝑓𝑏𝑒𝑂 𝑃𝑣𝑢 ! 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 𝐽𝑜 ? 𝑋𝑠𝑗𝑢𝑓𝑂 𝑊
𝑜𝑓𝑥
𝑃𝑣𝑢 ! 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 ∷= 𝑊
𝑜𝑓𝑥
𝑑𝑚𝑓𝑏𝑜𝑂 ∷= 𝐺𝑏𝑚𝑡𝑓 ¬ 𝑔𝑠𝑓𝑡ℎ𝑂 ∧ 𝑑𝑚𝑓𝑏𝑜𝑂 → 𝐸𝑝𝑥𝑜𝑂 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 ∷= 𝑡𝑓𝑠𝑤𝑓𝑠𝑊𝑏𝑚 𝑔𝑠𝑓𝑡ℎ𝑂 ∷= 𝑈𝑠𝑣𝑓
¬ 𝑑𝑚𝑓𝑏𝑜𝑂 → 𝑉𝑞𝑂 𝑑𝑚𝑓𝑏𝑜 ∷= 𝑈𝑠𝑣𝑓 𝑗𝑔 𝑔𝑠𝑓𝑡ℎ𝑂 𝑢ℎ𝑓𝑜 𝑗𝑔 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 ≠ 𝑡𝑓𝑠𝑤𝑓𝑠𝑊𝑏𝑚 𝑢ℎ𝑓𝑜 𝑔𝑠𝑓𝑡ℎ𝑂′ ∷= 𝐺𝑏𝑚𝑡𝑓 𝑔𝑝𝑠 𝑏𝑚𝑚 𝑂′ ≠ 𝑂 𝑡𝑓𝑠𝑤𝑓𝑠𝑊𝑏𝑚 ∷= 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 𝑓𝑚𝑡𝑓 𝑗𝑔 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂 ∉ 𝑡𝑓𝑠𝑤𝑓𝑠𝑊𝑏𝑚, ⊥ 𝑢ℎ𝑓𝑜 𝑑𝑝𝑜𝑔𝑚𝑗𝑑𝑢𝑡 ∷= 𝑑𝑝𝑜𝑔𝑚𝑗𝑑𝑢𝑡 ∪ { 𝑚𝑝𝑑𝑏𝑚𝑊𝑏𝑚𝑂}
[[ not(lookup(fresh,Node(N))) /\ lookup(clean,Node(N)) ]] =>> Down >-> dropBox [ In,Out,Down,Up ] ( serverVal , conflicts , update(localVal,Node(N),serverVal) , update(fresh,Node(N),True) , clean )
HIDE [Up,Down] IN dropbox NI
59
system model
TTCN TTCN
Test cases pass fail model-based test generation test execution
60
system model
pass fail
MBT
pass fail
61
dropbox model.txs
adapter adapter adapter
VM 0 Dropbox Client 0 VM 1 Dropbox Client 1
Dropbox Server
VM 2 Dropbox Client 2
TXS >> ..11: IN: In0 ! Write(Value("X")) TXS >> ..12: OUT: Out0 ! File(Value("SHK")) TXS >> ..13: IN: In2 ! Write(Value("A")) TXS >> ..14: OUT: Out2 ! File(Value("$")) TXS >> ..15: IN: In2 ! Write(Value("SP")) TXS >> ..16: OUT: Out2 ! File(Value("A")) TXS >> ..17: IN: In1 ! Write(Value("BH")) TXS >> ..18: OUT: Out1 ! File(Value("P")) TXS >> ..19: IN: In2 ! Read TXS >> ..20: OUT: Out2 ! File(Value("SP")) TXS >> ..21: IN: In0 ! Read TXS >> ..22: OUT: Out0 ! File(Value("X")) TXS >> ..23: IN: In2 ! Write(Value("PXH")) TXS >> ..24: OUT: Out2 ! File(Value("X"))
62
TXS >> ..77: IN: In0 ! Stabilize TXS >> ..78: OUT: Out0 ! File(Value("P")) TXS >> ..79: OUT: Out0 ! File(Value("L")) TXS >> ..80: OUT: Out0 ! File(Value("TK")) TXS >> ..81: OUT: Out0 ! File(Value("P")) TXS >> Expected: ( { Out0[ $"Out0"$1266 ] } , ( IF isFile($"Out0"$1266) THEN isValueInList(["PH","H"],value($"Out0"$1266)) ELSE False FI ) ) TXS >> FAIL: Out0 ! File(Value("P"))
63
model
pass fail TorXakis
https://github.com/TorXakis