Software Testing Software Testing CISC 323 Winter 2006 Prof. Lamb - PowerPoint PPT Presentation

Software Testing Software Testing CISC 323 Winter 2006 Prof. Lamb Prof. Kelly malamb@cs.queensu.ca kelly-d@rmc.ca

Included in Courseware • Reference material for the lectures – 19 pages from “Software Testing, A Craftsman’s Approach”, by Paul C. Jorgensen 2

All Testing is Sampling How can we sample in such a way that it is – Efficient – Effective • Another problem specifically for scientific and engineering software – Lack of accurate and detailed oracles • Often use the TLAR method (That Looks About Right!) Illustration of difficulties: next slides 3

4

5

6

Testing Techniques • Purpose – To provide reasoned ways of generating test cases – To produce test cases systematically • Might allow some sort of test case automation – To help in reproducibility of the tests – To address the”when are we done?” problem • Gives some level of confidence that enough bugs have been found – To allow better planning – To allow the use of different techniques to address different issues – To provide data for debugging 7

Testing Techniques • Black box testing – Also known as functional testing – Testing without opening the software box – We’ll look at examples of testing by considering the input data • White box testing – Also known as structural testing – Testing by considering what the code looks like – We’ll look at examples of code coverage 8

Boundary Value Testing (Black Box Testing) • Focuses on the boundaries of the input space to identify test cases – Based on the finding that errors seem to cluster at boundaries • Eg. code checks for <= instead of < • Assumes – software failures are rarely the result of the simultaneous occurrence of two or more faults • Test cases obtained by – Holding the values of all but one input variable at their nominal values and letting that one variable assume its extreme values • Minimum, just above minimum, nominal, just below maximum, maximum 9

Boundary Value Testing Example X 2 d c X 1 a b For a function of n variables, there are 4n+1 test cases 10

Robustness Testing • Extension of boundary value testing – See what happens when the extrema are exceeded • Value slightly greater than maximum • Value slightly less than minimum – Forces attention on exception handling • what happens or should happen when values fall outside their ranges 11

Weaknesses of Boundary Value and Robustness Testing • Assumption of independent failures • No consideration of interactions between input values • Meaning of the variables not taken into account • Example – {month, day} • (02,31) doesn’t make semantic sense, yet this may not be tested by this technique 12

Worst Case Testing • Reject the single fault assumption – Investigate what happens when more than one variable at a time is assigned an extreme value • For a function of n variables – Generates 5 n test cases • Limitations – No consideration of interactions between input values – Meaning of the variables not taken into account 13

Special Value Testing • Tester uses his/her domain knowledge to identify risky parts of the software system – Technique that is most dependent on the skills of the tester – No guidelines – use best engineering judgment – Can be used to “fill in” extra test cases not generated by any of the systematic techniques 14

Error in Courseware • Software Testing A Craftsman’s Approach • Page 62, Table 1 – What’s the goof? 15

Equivalence Class Testing (Black Box Testing) • Technique – Divide the input domain into classes • So that the requirements and/or design specifications require exactly the same behavior from each value in the class – Assumption • The program is constructed so that it either succeeds or fails for each of the values in that class – For robustness • For each equivalence class of valid inputs, we devise equivalence classes of invalid inputs 16

Equivalence Class Testing • Technique (cont’d) – Assume the equivalence classes are disjoint – Construct tests choosing one value from each appropriate equivalence class • Values are usually drawn at random each time the test is run – An equivalence class is considered covered if at least one value has been selected from it • There is research that says this is sufficient for effective testing – The goal is to cover all equivalence classes 17

Equivalence Class Testing Example – Registry of Your Pet • Fields: – Classification: • Dog – age, breed, colour, M/F, neutered • Cat – age, colour, M/F, neutered • Exotic: – Mammal: species, age, restricted, endangered, origin – Other: Phylum, Class, Order, Family, Genus, Species, restricted, endangered, origin 18

Equivalence Class Testing Example – Registry of Your Pet • Possible Valid Equivalence Classes – Restricted – Y/N – Classification: • Dog, Cat, Exotic – Endangered – Y/N – Age: – Origin • 0-200 • Antarctica, Algeria, – Breed Australia, Brazil, … • Dalmatian, boxer, poodle, … – Phylum – alpha (0-40) – Colour – Class – alpha (0-40) • Black, white, brown, gray, calico, – Order – alpha (0-40) tiger, … – M/F – Family – alpha (0-40) – Neutered – Y/N – Genus – alpha (0-40) – Exotic – Mammal, other – Species other – alpha (0-40) – Mammal Species • Corresponding Invalid • Monkey, ape, tiger, cheetah, Equivalence Classes …? lemur, wolf, raccoon, harbour seal, … 19

Equivalence Class Testing • Limitations – assumes all values within an equivalence class are equally likely to cause failure – assumes the implementation of the class conforms to its specification • If the implementation does not match the requirement/design as specified (which it will not if a defect exists), then some values in the equivalence class will produce a different actual behavior than expected – Those values may not be chosen during equivalence class testing 20

Equivalence Class Testing • Controversy whether random values drawn from the whole input space are as effective at finding defects as are random values chosen based on equivalence classes 21

Testing Techniques based on Code Coverage (White Box) • Statement coverage – Touch each statement once • Branch coverage – Execute each branch after a condition statement • Cause each condition to be set to true, then set to false • Path coverage – Execute all paths through the code • Generally infeasible in all but simplest cases • Basis path coverage – A compromise 22

Example of Statement, Branch, and Path Coverage void tst(int x) { if (x>0) pos = pos+1; if (x%2==0) even = even+1; return; } • Tests required for: – Statement Coverage: tst(2) – Branch Coverage: tst(-1),tst(2) – Path Coverage: tst(-2),tst(-1),tst(1),tst(2) 23

Use of Statement Coverage for “When are we done?” Problem • Weakest type of the three coverage types (statement, branch, path) as far as effective test case generation • Easiest to support with a tool – Many tools available to track statement coverage • Tools add “instrumentation” to your code to track which statements have been executed – Could slow your code down unacceptably • After all your tests are run, tools report on total code coverage – Often used to decide adequate testing • Even for this, 100% coverage may not be feasible – May have special conditions, dead code, error routines • Best use is to find what parts of the code have not been tested at all – It may be more cost effective to inspect uncovered parts than it is to cover them by testing 24

100% Coverage? • Even 100% coverage does not guarantee that software is fault free or reliable • The Law of Diminishing Returns – The more a program has been tested, the less a given test set can further contribute to the test adequacy. 25

Example Code with #include <stdio.h> two Branches (part of #include <stdlib.h> #include <math.h> Myers Triangle) main(argc, argv) int argc; char* argv[]; { int sideA; int sideB; int sideC; double s; double Area; sideA = atoi(argv[1]); sideB = atoi(argv[2]); sideC = atoi(argv[2]); if ( (sideA == sideB) && (sideA == sideC ) ) { s = 0.5 * (sideA + sideB + sideC); Area = sqrt (s / (s - sideA) * (s - sideB) * (s - sideC) ); printf ( "area = %g\n", Area) ; } Else { puts ("not an equilateral triangle") ; return 0; } 26

Branch Coverage: 2 test cases needed • To exercise the TRUE • To exercise the FALSE branch branch – use inputs – use inputs • Side A = 2 • Side A = 3 • Side B = 2 • Side B = 4 • Side C = 2 • Side C = 5 – output – output • area = 1.73205 • not an equilateral triangle 27

Two Errors not Caught by Test Cases • wrong operand to read in sideC – should be argv[3] • Heron’s formula for area of a triangle – Area = [s (s - sideA) (s - sideB) (s - sideC) ] – code has division operator instead of multiplication – 2 is the only value for which the wrong equation gives the right answer 28