Model-based Testing Part 1 Prof. Dr.-Ing. Ina Schaefer SFM:ESM - - PowerPoint PPT Presentation

Platzhalter für Bild, Bild auf Titelfolie hinter das Logo einsetzen Model-based Testing – Part 1 Prof. Dr.-Ing. Ina Schaefer – SFM:ESM - Bertinoro - 18 June 2014

Contents Part 1: Foundations of Testing and Model-based Testing  Fundamental Notions and Concepts of Software Testing  Model-based Testing A Theoretical Perspective on Model-based Testing  Part 2: Model-based Testing of Software Product Lines  Sample-based Software Product Line Testing  Regression-based Software Product Line Testing  Variability-Aware Software Product Line Testing June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 2

Testing is … […] ¡“an ¡activity ¡performed ¡for ¡evaluating ¡product ¡quality, ¡and ¡ for ¡improving ¡it, ¡by ¡identifying ¡defects ¡and ¡problems.” […] ¡“the ¡process ¡of ¡operating ¡a ¡system ¡or ¡component ¡under ¡ specified conditions, observing or recording the results, and making an evaluation or some aspects of the system or component.” [IEEE, 1990] June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 3

Software Testing is … […] ¡“an ¡activity ¡for ¡checking ¡or ¡measuring ¡some ¡quality ¡ characteristics of an executing object by performing experiments ¡in ¡a ¡controlled ¡way ¡w.r.t. ¡a ¡specification.” [Tretmans, 1999] June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 4

Factors for Testing test aims test methods • static testing: • functional e.g. systematic code inspections • non-functional • dynamic testing: e.g. experimental executions • robustness Tester • performance SUT Specification • reliability June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 5

Factors for Testing information base test scale • unit tests • black box • component tests public class foo { p boolean bar() { if (weekend) { return true ; • white box • integration tests } else { return false ; } } } • system tests public class foo public { public { boolean bar() if (weekend) { • grey box return true ; } else { return false ; } } } June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 6

Dynamic Software Testing Tester Platform Environment I System Specification under Test (SUT) O Test Case Design Test Case Execution June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 7

Failure, Fault, Error failure - A failure is an undesired observable behavior of an SUT. fault – A fault in an SUT causes a failure during test execution. error – An error is a logical flaw in the implementation. June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 8

The Notion of Software Testing used in this Lecture Software testing consists of the dynamic validation/verification of the behavior of a program on a finite set of test cases suitably selected from the usually infinite input domain against the expected behavior. June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 9

Some Literature on Software Testing • Myers, G.J.: The Art of Software Testing. Wiley, New York • Beizer, B.: Software Testing Techniques. Van Nostrand Reinhold Co. • Broy; M. (ed.): Model-Based Testing of Reactive Systems: Advanced Lectures. Springer, Berlin Heidelberg • IEEE: Standard Glossary of Software Engineering Technology 610.121990 • IEEE: Standard for Software Test Documentation Std. 829-2008 • van Veenendaal, E. (ed.): ISTQB Glossary of Testing Terms 2.2. Glossary Working Party June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 10

Contents Part 1: Foundations of Testing and Model-based Testing  Fundamental Notions and Concepts of Software Testing  Model-based Testing A Theoretical Perspective on Model-based Testing  Part 2: Model-based Testing of Software Product Lines  Sample-based Software Product Line Testing  Regression-based Software Product Line Testing  Variability-Aware Software Product Line Testing June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 11

Model-Based Testing Tester Platform Environment I Specification SUT O Test Case Design Test Case Execution June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 12

Model-Based Testing model-based system test test model model generation Platform Environment I test SUT execution O pass/fail June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 13

Model-based Testing Model-based testing is the automation of black box tests. abstract test case test case concretization centered around test model selection model-based system test test model model generation basis for test case selection Platform Environment and coverage measure I blackbox test SUT execution O result verdict w.r.t. dynamic observations expected behavior pass/fail June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 14

Contents Part 1: Foundations of Testing and Model-based Testing  Fundamental Notions and Concepts of Software Testing  Model-based Testing  A Theoretical Perspective on Model-based Testing Part 2: Model-based Testing of Software Product Lines  Sample-based Software Product Line Testing  Regression-based Software Product Line Testing  Variability-Aware Software Product Line Testing June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 15

MBT from a Theoretical Point of View 𝒋 ≃ 𝒕 with implemetation i and formal behavioral implementation relation: specification s 𝒋 ⊑ 𝒕 implementation shows at most the behaviors of the preorder relation: specification 𝒋 ¡𝒅𝒑𝒐𝒈𝒑𝒔𝒏𝒕 ¡𝒕 ∶⇔ ¡ 𝒋 ⊆ 𝒕 intentional conformance: where ⋅ ¡ defines sets of all observable behaviors 𝒋 ¡𝒅𝒑𝒐𝒈𝒑𝒔𝒏𝒕 ¡𝒕 ∶⇔ ∀𝒗 ∈ 𝓥 ∶ 𝒑𝒄𝒕 𝒗, 𝒋 ≈ 𝒑𝒄𝒕(𝒗, 𝒕) extensional conformance: where 𝒱 defines sets of all observers June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 16

Model-based I/O Conformance Testing • Proposed by Jan Tretman in ¡the ¡90’s • Model-based functional conformance testing of systems with reactive, non-deterministic behaviors • Input, output, and quiescence based testing theory • Based on I/O labeled transition systems as test models AND implementation models • Proven sound and exhaustive • Rich tool support • Formal basis for many advanced testing frameworks June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 17

Running Example Beverage vending machine • Input actions 𝐽 = 1€, 2€ • • Transitions ¡labels ¡prefixed ¡with ¡“?” • Output actions 𝑉 = {𝑑𝑝𝑔𝑔𝑓𝑓, 𝑢𝑓𝑏} • • Transition ¡labels ¡prefixed ¡with ¡“!” June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 18

I/O-Labeled Transition Systems I/O Labeled Transitionsystem: 𝑹, 𝒓 𝟏 , 𝑱, 𝑽, → , where • 𝑹 is a countable set of states, 𝒓 𝟏 ∈ 𝑹 is the initial state, • 𝑱 and 𝑽 are disjoint sets of input actions and output actions, and • → ¡⊆ 𝑹 × 𝒃𝒅𝒖 × 𝑹 ¡ is a labeled transition relation. • June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 19

LTS - Examples 𝑟 � ? ¡1€ 𝑈𝑠 𝑟 � = {? 1€, ? 1€ ⋅ ! 𝑑𝑝𝑔𝑔𝑓𝑓, ? 1€ ⋅ ! 𝑢𝑓𝑏} ! 𝑑𝑝𝑔𝑔𝑓𝑓 ! 𝑢𝑓𝑏 𝑟 � 𝑈𝑠 𝑟 � = ? 1€, ? 2€ ? ¡1€ ? 2€ June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 20

LTS Trace Semantics Each compution refers to some path � � 𝑡 � � � 𝑡 � � � ⋅⋅⋅ � ��� 𝑡 ��� � � 𝑡 � 𝑟 � The behavior of a computation is defined by a trace 𝑢𝑠𝑏𝑑𝑓 ¡𝜏 = 𝜈 � 𝜈 � ⋅⋅⋅ 𝜈 � ∈ 𝑏𝑑𝑢 ∗ June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 21

LTS - Examples 𝑟 � ? ¡1€ ? 2€ 𝑈𝑠 𝑟 � = ? 1€, ? 2€, ? 1€ ⋅ ! 𝑑𝑝𝑔𝑔𝑓𝑓, ? 2€ ⋅ ! 𝑑𝑝𝑔𝑔𝑓𝑓 ! 𝑑𝑝𝑔𝑔𝑓𝑓 𝑟 � ? ¡1€ ? 2€ 𝑈𝑠 𝑟 � = ? 1€, ? 2€, ? 1€ ⋅ ! 𝑑𝑝𝑔𝑔𝑓𝑓, ? 1€ ⋅ ! 𝑢𝑓𝑏, ? 2€ ⋅ ! 𝑑𝑝𝑔𝑔𝑓𝑓, ? 2€ ⋅ ! 𝑢𝑓𝑏 𝜐 𝜐 ! 𝑑𝑝𝑔𝑔𝑓𝑓 ! 𝑢𝑓𝑏 June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 22

LTS Trace Notations Let s be an I/O 𝓜𝓤𝓣 , 𝝂 𝒋 ∈ 𝑱 ¡ ∪ 𝑽 ∪ 𝝊 and 𝒃 𝒋 ∈ 𝑱 ∪ 𝑽 ¡ . 𝑡 � � � ⋅⋅⋅� � 𝑡 � ≔ ¡∃𝑡 � , … , 𝑡 � ∶ 𝑡 = 𝑡 � � � 𝑡 � � � ⋅⋅⋅ � � 𝑡 � = 𝑡 � 𝑡 𝑡 � � � ⋅⋅⋅� � ≔ ¡∃𝑡′ ∶ 𝑡 = 𝑡 � � ⋅⋅⋅� � 𝑡 � 𝑡 … � � ⋅⋅⋅� � ¡≔ ∄𝑡 � ∶ 𝑡 � � ⋅⋅⋅� � 𝑡′ ¬𝑡 𝑡 � June 18th, 2014 | Ina Schaefer | Model-based Testing – Part 1 | Seite 23