Model-Based Testing There is Nothing More Practical than a Good - PowerPoint PPT Presentation

Model-Based Testing There is Nothing More Practical than a Good Theory model Jan Tretmans SUT ESI – Embedded Systems Innovation by TNO Radboud University Nijmegen Högskolan i Halmstad pass fail jan.tretmans@tno.nl

Jan Tretmans Radboud University Nijmegen The Netherlands ESI Embedded Systems Innovation UT (TNO) Eindhoven RU The Netherlands ESI

Overview : Model-Based Testing 1. Introduction – Motivation – Basics: what is it – Context: Model-Based Verification, Validation, Code Generation, . . . 2. Theory MBT with Labelled Transition systems – models – implementation relation ioco – testing for ioco 3. Tools – TorXakis - a tool voor ioco testing – Representation of models : Language 4. Applications – dropbox 3

Model-Based Testing Motivation 4

Trends & Challenges complexity size connectivity systems-of-systems multi disciplinarity Model Based Testing change variability heterogeneous evolvability components uncertainty 5

MBT : Next Step Challenges concurrency abstraction parallelism complexity statistical state + usage complex data profiles connectivity multi disciplinarity Model model under composition Based specification Testing change test variability partial heterogeneous selection specification evolvability components uncertainty multiple uncertainty paradigms nondeterminism integration 6

Model-Based Testing Basics 7

Software Testing specification-based, Checking or measuring active, black-box testing of some quality characteristics functionality of an executing software object by performing experiments in a controlled way w.r.t. a specification tester SUT specification System Under Test 8

1 : Manual Testing 1. Manual testing SUT System Under Test pass fail 9

2 : Scripted Testing test TTCN TTCN cases 1. Manual testing 2. Scripted testing test SUT execution pass fail 10

3 : Keyword-Driven Testing high-level test notation test scripts 1. Manual testing 2. Scripted testing test SUT execution 3. Keyword-driven testing pass fail 11

4 : Model-Based Testing model-based Test system test TTCN TTCN cases generation model 1. Manual testing 2. Scripted testing test SUT execution 3. Keyword-driven testing 4. Model-based testing pass fail 12

!coffee MBT : Example Models ?button !alarm ?coin ?button 13

MBT : Example Models

MBT : Benefits MBT: next step in test automation model-based test • model Automatic test generation generation + test execution + result analysis • More, longer, and diversified test cases more variation in test flow and in test data test SUT execution • Model is precise and consistent test basis unambiguous analysis of test results pass fail • Test maintenance by maintaining models improved regression testing • Expressing test coverage detecting more bugs model coverage faster and cheaper customer profile coverage

Model-Based Verification, Validation, Testing, . . . . . 16

Doing Something with Models • Modelling making a model reveals errors • Simulation go step-by-step through the model • Model checking go through all states of the model • Theorem proving prove theorems about the model • Code generation executable code from the model • Testing test an implementation for compliance • Model learning generate a model from observation 17

Validation, Verification, Testing informal requirements informal world validation verification formal world model model-based testing real world SUT 18

Code Generation from a Model A model is more ( less ) than code generation: • views • abstraction • testing of aspects • verification and validation of aspects 19

Code Generation from a Model model of 𝒚 ? x (x >= 0) ! y y x y = x • specification of properties rather than construction • under-specification • non-determinism 20

Spectrum of Models abstract (test) models design models virtualization realization 21

Model Based Testing model-based Test system test TTCN TTCN cases generation model SUT conforms to model test SUT execution pass fail 22

Testing with Models tester software system mechanical physical environment 23

Testing with Models software tester model software system mechanical physical model environment model 24

A Theory of Model-Based Testing with Labelled Transition Systems 25

MBT with LTS Models 26

Models: Labelled Transition Systems Labelled Transition System:  S, L I , L U , T, s 0  initial state s 0  S states transitions input T  S  (L  {  })  S output ? = input labels labels ! = output !coffee ?button !alarm ?coin ?button 27

MBT with LTS ioco 28

MBT : Labelled Transitions Systems ioco Test LTS test TTCN TTCN cases model generation SUT ioco model input/output sound   set of conformance  LTS tests exhaustive ioco SUT LTS SUT passes tests test behaving as execution input-enabled LTS pass fail 29

Input/Output Conformance : ioco = def    Straces (s) : out (i after  )  out (s after  ) i ioco s s is a Labelled Transition System i is (assumed to be) an input-enabled LTS  =  !x  L U  {  } . p !x p p  Straces ( s ) = {   ( L  {  } )* | s }  p after  = { p’ | p p’ }  out ( P ) = { !x  L U | p , p  P }  {  | p !x p , p  P } 30

Example: ioco ?quart ?dime ?dime ?dime ?quart specification ! choc !tea ! coffee model ?dime ?quart ?dime ! coffee !tea ?dime ?quart ?dime ?dime  ! choc !tea ! coffee non-determinism uncertainty under-specification 31

MBT : Nondeterminism, Underspecification ? x (x < 0) specification model SUT models ? x (x >= 0) ? x (x >= 0) !  x ! y ? x ( | y x y – x| < ε ) ? x (x < 0) • non-determinism !error • under-specification ? x (x >= 0) • specification of properties ? x ! -  x rather than construction 32

MBT with LTS Testing for ioco 33

Test Case ?coffee !dub ?tea test case = labelled transition system fail fail ?coffee !kwart ?tea – ‘quiescence’ / ‘time - out’ label  pass fail  – tree-structured ?coffee ?tea – finite, deterministic – final states pass and fail fail fail ?coffee – from each state  pass , fail : !dub ?tea fail • either one input !a fail • or all outputs ?x and  ?coffee  ?tea pass pass fail 34

Test Generation Algorithm : ioco Algorithm to generate a test case t(S) from a transition system state set S, with S   ( initially S = s 0 after  ). Apply the following steps recursively, non-deterministically: 1 end test case 3 observe all outputs pass forbidden outputs allowed outputs ?y ?x  2 supply input !a allowed outputs fail fail forbidden outputs ?y ?x !a t ( S after !x ) fail fail allowed outputs (or  ): !x  out (S) t ( S after !x ) forbidden outputs (or  ): !y  out ( S ) t ( S after ?a   ) 35

Example: ioco Test Generation s t specification generated model test case !dime ?dime ! coffee !tea ?tea ?choc  ?coffee pass pass fail fail i ioco s i implementation ?dime i fails t ! choc !tea i ioco s = def    Straces (s) : out (i after  )  out (s after  ) 36

MBT with ioco is Sound and Exhaustive Test assumption :  SUT  IMP .  m SUT  IOTS .  t  TESTS . gen : LTS →  (TTS) s  LTS SUT passes t  m SUT passes t Prove soundness and exhaustiveness: i ioco s  m  IOTS . (  t  gen( s ) . m passes t ) SUT test  t  SUT behaving as m ioco s tool input-enabled LTS SUT comforms to s   pass fail sound exhaustive SUT passes gen(s) 37

TorXakis A Tool for MBT with LTS 38

MBT Tools ioco • • • M-Frame tedeso AETG • • • MISTA Temppo Agatha • • • NModel TestGen (Stirling) Agedis • • • OSMO TestGen (INT) Autolink • • • ParTeG TestComposer Axini Test Manager • • • Phact/The Kit TestOptimal Conformiq • • • PyModel TGV Cooper • • • QuickCheck Tigris Cover • • • Reactis TorX DTM • • • Recover TorXakis fMBT G  st • • • RT-Tester T-Vec • • • SaMsTaG Tveda Gotcha • • • Smartesting CertifyIt Uppaal-Cover Graphwalker • • • Spec Explorer Uppaal-Tron JTorX • • • . . . . . . . . . . . StateMate MaTeLo • • STG MBTsuite 39