Formal Testing of Formal Testing of g Distributed Systems Distributed Systems R. M. Hierons R. M. Hierons B B Brunel University, UK Brunel University, UK l U i l U i it it UK UK rob.hierons@brunel.ac.uk rob.hierons@brunel.ac.uk http://people.brunel.ac.uk/~csstrmh http://people.brunel.ac.uk/~csstrmh Networked and Distributed Networked and Distributed Systems Systems
Work With Work With � Mercedes Mercedes Merayo Merayo � Manuel Nunez Manuel Nunez � Jessica Chen Jessica Chen � Hasan Hasan Ural Ural Networked and Distributed Networked and Distributed Systems Systems
Challenges in Testing Challenges in Testing � These include: These include: � Scale Scale � Concurrency Concurrency � Distribution Distribution � The oracle problem. The oracle problem. � Potential solution, model Potential solution, model-based testing: based testing: � Automate testing on the basis of a formal Automate testing on the basis of a formal model or specification. model or specification. Networked and Distributed Networked and Distributed Systems Systems
Model Based Testing Model Based Testing � We only observe interactions between the We only observe interactions between the W W l l b b i t i t ti ti b t b t th th system under test (SUT) and its environment. system under test (SUT) and its environment. � To reason about test effectiveness we To reason about test effectiveness we assume: assume: • The behaviour of the SUT can be expressed in the The behaviour of the SUT can be expressed in the same language as the model. same language as the model. Networked and Distributed Networked and Distributed Systems Systems
Models for distributed and Models for distributed and networked systems networked systems � Such systems typically: Such systems typically: � Have states and actions Have states and actions � Are concurrent Are concurrent � If we take a black If we take a black-box view, the last issue box view, the last issue is less important is less important is less important is less important Networked and Distributed Networked and Distributed Systems Systems
Formal languages Formal languages � Typically use states and transitions Typically use states and transitions between states triggered by actions. between states triggered by actions. b t b t t t t t t i t i d b d b ti ti � Many can be seen as one of: Many can be seen as one of: � Finite state machines Finite state machines � Labelled Labelled transition systems (and input output transition systems (and input output transition systems) transition systems) transition systems) transition systems) � Former less general but the models are Former less general but the models are easier to easier to analyse analyse. Networked and Distributed Networked and Distributed Systems Systems
Multi Multi-port systems port systems � Physically distributed interfaces/ports. Physically distributed interfaces/ports. � A tester at each port. A tester at each port. tester tester SUT tester Networked and Distributed Networked and Distributed Systems Systems
Distributed testing Distributed testing � Mainly focus on the simplest approach: Mainly focus on the simplest approach: � The testers cannot communicate with one The testers cannot communicate with one another another � There is no global clock There is no global clock � Observations are ‘local’ Observations are ‘local’ Networked and Distributed Networked and Distributed Systems Systems
Motivation Motivation � Initially just testing/test generation. Initially just testing/test generation. � The discussion will be around both Th Th The discussion will be around both di di i i ill b ill b d b th d b th • testing testing and and • implementation/conformance relations implementation/conformance relations . � Testing from: Testing from: • input output transition systems and possibly input output transition systems and possibly � deterministic finite state machines d t deterministic finite state machines d t i i ti fi it i i ti fi it t t t t hi hi � nondeterministic finite state machines nondeterministic finite state machines Networked and Distributed Networked and Distributed Systems Systems
Testing and Testing and Testing and Testing and Observations Observations Networked and Distributed Networked and Distributed Systems Systems
Global Traces Global Traces � A global trace is a sequence of inputs and A global trace is a sequence of inputs and outputs outputs outputs. outputs. � We assume there are m ports and: We assume there are m ports and: � x p will denote an input at port p (from X will denote an input at port p (from X p ) ,...,y m ) ∈ Y, Y= (Y 1 ∪ {-}) × … × (Y (Y m ∪ {-}), will be � (y (y 1 ,...,y Y, Y=(Y }), will be an output an output A global trace is an element of (X × Y)* � A global trace is an element of (X A A l b l t l b l t i i l l t t f (X f (X Y)* Y)* Y)* Networked and Distributed Networked and Distributed Systems Systems
Consequences Consequences � Each tester observes only the interactions ( Each tester observes only the interactions ( local local trace trace ) at its port trace trace ) at its port ) at its port ) at its port Tester 1 SUT Tester 2 x 1 y 1 y 2 x 1 y 1 � The tester at port 1 observes x The tester at port 1 observes x 1 y 1 x 1 y 1 and the and the tester at 2 observes y tester at 2 observes y 2 only. only. Networked and Distributed Networked and Distributed Systems Systems
What the testers observe What the testers observe � Given global trace z, the tester at p Given global trace z, the tester at p observes a local trace π p (z) . observes a local trace b b l l l t l t ( ) ( ) (z) . Tester 1 Tester 2 x 1 y 1 y 2 x 1 y 1 Networked and Distributed Networked and Distributed Systems Systems
Controllability problems Controllability problems � The following test has a controllability The following test has a controllability problem: introduces nondeterminism into problem: introduces nondeterminism into bl bl i t i t d d d t d t i i i i i t i t testing. testing. tester SUT tester Networked and Distributed Networked and Distributed Systems Systems
Observability Observability problems problems � The following look the same The following look the same tester Spec tester tester SUT tester x 1 x 1 y 1 y 1 y 2 x 1 x 1 1 y 2 y 1 y 1 � Testers/users cannot ‘map’ output to input Testers/users cannot ‘map’ output to input Networked and Distributed Networked and Distributed Systems Systems
Equivalent global traces Equivalent global traces � Since we only observe local traces: Since we only observe local traces: � Global traces z and z’ are indistinguishable if Global traces Global traces Global traces z and z’ are indistinguishable if and ’ are indisting ishable if and ’ are indisting ishable if their projections are identical: the local traces their projections are identical: the local traces are the same. are the same. We denote this: z ∼ z’ � We denote this: z z’ The following are equivalent under ∼ � The following are equivalent under � x 1 /(y x /(y /(y y )x /(y 1 ,y ,y 2 )x )x 1 /(y )x /(y /(y /(y 1 ,-) ) � x 1 /(y /(y 1 ,-)x )x 1 /(y /(y 1 , y , y 2 ) � Both have x Both have x 1 y 1 x 1 y 1 at port 1 and y at port 1 and y 2 at 2. at 2. Networked and Distributed Networked and Distributed Systems Systems
Problem: Test effectiveness is not Problem: Test effectiveness is not monotonic monotonic � Example: x Example: x 1 detects a fault but x detects a fault but x 1 x 1 does does not. not. t tester SUT tester tester Spec tester x 1 x 1 y 1 y 1 y 2 x x 1 x 1 y 2 y 1 y 1 Networked and Distributed Networked and Distributed Systems Systems
Two approaches to defining Two approaches to defining implementation relations implementation relations � We might have: We might have: � Agents at ports are entirely ‘independent’: Agents at ports are entirely ‘independent’: • No external agent can receive information No external agent can receive information regarding observations at more than one port regarding observations at more than one port � Or the local traces observed at the ports can Or the local traces observed at the ports can be ‘brought together’ later. be ‘brought together’ later. Networked and Distributed Networked and Distributed Systems Systems
Differences Differences � Specification Specification Tester 1 Tester 2 Tester 1 Tester 1 Tester 2 Tester 2 x x y y' z z' � SUT SUT Tester 1 SUT Tester 2 x y' z Networked and Distributed Networked and Distributed Systems Systems
Using an external network Using an external network � If we connect the testers using an external If we connect the testers using an external network, network, sometimes t t k k sometimes we can overcome ti ti we can overcome controllability and observability problems. controllability and observability problems. tester SUT tester tester SUT tester Networked and Distributed Networked and Distributed Systems Systems
Recommend
More recommend
