Black Box Testing (A&O Ch. 4) (2 nd Ed. Ch. 6) Course Software - PowerPoint PPT Presentation

Black Box Testing (A&O Ch. 4) (2 nd Ed. Ch. 6) Course Software Testing & Verification 2019/20 Wishnu Prasetya & Gabriele Keller

Plan • Partitioning based testing (A&O Ch 4/2 nd Ed Ch. 6). • Model based testing Note: black box testing is an important concept. For example, system level testing is often done in a black box setup. In this lecture we will discuss two important techniques that are commonly used in such a setup. The first one, partition based testing, is discussed in length in A&O. However, the second one, model based testing (MBT), is only lightly touched in A&O. This lecture will introduce you to some basic concepts of MBT. We will borrow some key concepts from Hoare’s “Communicating of Sequential Processes” (CSP). For a more practical exposition on MBT you can look at e.g. Practical Model-Based Testing by Utting - Lageard (UU students can access it for free from Scienc Direct, https://www.sciencedirect.com/science/book/9780123725011) 2

White box testing Testing is “ white box testing ” if you have knowledge of the source code of the target program to help you designing the tests, and your tests have in principle access to all the program’s variables so that they can inspect them if they are in the correct state. • Compare this with Def 1.26 A&O (2 nd Ed. See p 26). • White box setup is common at the unit-testing level. 3

Black box testing Testing is ” black box testing ” if you do not assume full knowledge of the inner working of the target program. Usually this also entails that your tests have only limited access to the program state to inspect its correctness. Compare with Def 1.25 in A&O (2 nd Ed. See p26). • Note that although we may have full access to the source code, at the system level • we may choose not to use this knowledge because it becomes to complicated to comprehend. Black box testing is common at the system-testing level. We will discuss two approaches : (1) Partition-based testing , Ch 4 (2 nd ed. Ch. 6) and (2) Model-based testing . 4

Partition-based testing • Premise: The input space of a program can be partitioned into “equivalence classes”, such that inputs from the same partition/equivalence-class lead to the “same kind” of output/effect. • It then makes sense to require that every partition should be tested at least once. • Even without source code, we can often propose a reasonable partitioning. 5

Recall this example... Enum TriType { Isosceles, Equilateral, Scalene } TryType triangle(Float a, Float b, Float c) { ... } If a, b, c represent the sides of a triangle, this methods determines the type of the triangle. Let’s now try to come up with test cases for this function from the black box perspective. 6

Partitioning triangle()’s domain ISOSCELES SCALENE EQUIDISTANCE NON TRIANGLE input domain of triangle This suggests 4 test-cases, one for every partition. 7

Boundary value test Errors often lurk in the “boundaries” between partitions à test values on and around the boundaries. Examples: triangle(1,2,2.999) ISOLECES SCALENE EQUIDISTANCE NON TRIANGLE triangle(1,2,3) 8

Example 2 int incomeTax (int income, int age, int children) Fragments from its informal specification: the method calculates income tax: • Income below 10K is not taxed. • A person under 18y or above 70y is not taxed, as long as the income is below 50K. • Tax reduction applies, linear to the number of children the person has. This method has a more complex input space than the “triangle” example. 9

Consider the following partitioning The input domain of incomeTax(..) <10k <18 0 X X [18..70] [10k..50k) >0 >70 ≥50k age #children income • As before, we can try to come up with test cases that would cover every partition. Note that just 3 test cases can cover all partitions! • à does not feel very strong. This is because the approach ignores that different parameters may ”interact”. E.g. normally when the age<18 the person will not be taxed. However, the combination of income≥50k and age<18 will trigger its own behavior, namely that the person will be taxed anyway. 10

Combinatoric testing incomeTax(income,age,#children) <10k <18 0 X X [18..70] blocks [10k..50k) >0 >70 ≥50k age income #children characteristics In this example the input domain is spanned by 3 characteristics, • and there are in total 8 blocks. A block combination over the characteristics, e.g. (<10k,<18,0) • abstractly specifies a test case. There are in total 3x3x2 = 18 such combinations of blocks. • 11

Combinatoric testing Let’s name the blocks for A A A convenience: X X B B B C C income age #children • Stats: 8 blocks, 18 combinations. • (C4.24/2 nd Ed. C6.2, EACH CHOICE coverage) Each block must be tested. |T| = ( max i : 0 £ i < k : B i ) ; usually too weak. • (C4.23/2 nd Ed. C6.1, ALL coverage) All combinations must be tested. | T | = ( P i : 0 £ i < k : B i ) ; does not scale up. 12

How about to just cover all pairs? A A A X X B B B C C income age #children Premise: most of the times, only t out of N parameters actually • influence the behavior. E.g: – when the income is less than 10k, there will be no tax, regardless the age and #children. – When the income is ≥50k, there will be tax, regardless the age. If we speculate on t =2, this leads to “pair-wise” testing: to cover all • block-pairs over different characteristics. 13

Pair-wise and t-wise testing A A A X X B B B C C income age #children (C4.25/2 nd Ed. C6.3, pair-wise coverage ). Each pair of blocks (from • different characteristics) must be tested. There are 9+6+6 = 21 pairs to cover. But we can cover them with just • 9 test cases. Pair-wise coverage is stronger than EACH CHOICE, and still scalable. • (C4.26/2 nd Ed. C6.4, t-wise coverage ). Generalization of pair-wise. • 14

Example: a test set with full pair-wise coverage A A A X X B B B C C income age #children income age #children Test-case 1 A A A Test-case 2 A B B Test-case 3 A C A Test-case 4 B A B Test-case 5 B B A Test-case 6 B C B Test-case 7 C A A Test-case 8 C B B Test-case 9 C C Does’t matter 15

PWC, example 2 • Four characteristics, each with two blocks: A A A A B B B B • Minimal test set that gives PWC: cha3 cha4 cha1 cha2 tc1 A A A B A tc2 B B A B tc3 A B B tc4 B A B B tc5 A A B A 16

Consider again this test set <18 <10k A A 0 A X X [18..70] [10k..50k) B B B >0 ≥50k C C >70 income #children age income age #children • Despite giving full pwc these tests tc1 A A A miss some important cases: tc2 A B B – We want to fully test (B,B,x) à tc3 A C A missing (B,B,B). tc4 B A B – We want to fully test (C,x,x) à tc5 B B A missing e.g. (C,A,B) and (C,C,B) tc6 B C B • Solution: add constraints, but... tc7 C A A (see next slide). tc8 C B B tc9 C C A 17

Pair-wise and t-wise testing • Consider a program P with N characteristics [1.. N ]. For simplicity, suppose each characteristic is split into B number of blocks. • Total number of pairs = B 2 * V 2 where V 2 is the number of subsets of size 2 out of N characteristics. So, V 2 = 𝑂 !! 2 = #!∗ !%# ! • To cover all pairs you need at least B 2 number of test cases. Expect that it can be more (as in Example 2). • In general, finding a minimum size test set that gives full t-wise coverage is not trivial. • However, as pointed out before, k-wise testing ignores the “semantic” (that some combinations should be included or excluded (because they are not sensical)). We can add constraints, though this makes the problem of calculating the minimum test set even harder. 18

Adding a bit of semantic Example : t 0 = ( B,Q,X ), generates these A P X additional test requirements : B Q Y (B,Q,Y) C R (A,Q,X) (C,Q,X) (B,P,X) (B,R,X) (C4.27/2 nd Ed. C6.5, Base Choice Coverage , BCC) Decide a single base test t 0 . Make more tests by each time removing one block from t0, and forming combinations with all remaining blocks (of the same characteristics). | T | = 1 + ( S i : 0 £ i < k : B i - 1) 19

Consider again Base test t 0 = ( B,Q,X ), and these A P X additional test, giving full BCC: B Q Y (B,Q,Y) C R (A,Q,X) (C,Q,X) (B,P,X) (B,R,X) • What if we also need to insist on testing out all combinations of (C,P,-) ? • Proposal: use multiple “base tests” 20

MBCC Decide the “base blocks”: (red) A P X Choose one or more best tests. They can only B Q Y use base blocks. E.g.: C R base test t 0 = ( B,Q,X ) base test t 1 = ( C,P,X ) (C4.28/2 nd Ed. C6.6, Multiple Base Choices coverage). For each characteristic we decide at least one base block . Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remaining non-base blocks. | T | at most M + M*( S i : 0 £ i < k : B i - m i ) 21

Example MBCC The base blocks are marked red. The base A P X tests: B Q Y base test t 0 = ( B,Q,X ) C R base test t 1 = ( C,P,X ) We need to add these tests to get full MBCC: • Varying t 0 over non-base blocks: (A,Q,X), (B,R,X), (B,Q,Y) • Varying t 1 over non-base blocks: (A,P,X), (C,R,X), (C,P,Y) • Remove duplicates (there are none above) 22