Learning objectives Understand the purpose of defining test - PowerPoint PPT Presentation

Learning objectives • Understand the purpose of defining test adequacy criteria, and their limitations Test Case Selection • Understand basic terminology of test selection and Adequacy Criteria and adequacy • Know some sources of information commonly used to define adequacy criteria • Understand how test selection and adequacy criteria are used (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 1 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 2 Adequacy: We can’t get what we want Adequacy Criteria as Design Rules • Many design disciplines employ design rules • What we would like: – E.g.: “traces (on a chip, on a circuit board) must be – A real way of measuring effective testing at least ___ wide and separated by at least ___” If the system system passes an adequate suite of – “The roof must have a pitch of at least ____ to shed test cases, then it must be correct (or dependable) snow” • But that’s impossible! – “Interstate highways must not have a grade greater – Adequacy of test suites, in the sense above, is than 6% without special review and approval” provably undecidable. • Design rules do not guarantee good designs • So we’ll have to settle on weaker proxies for – Good design depends on talented, creative, disciplined designers; design rules help them avoid adequacy or spot flaws – Design rules to highlight inadequacy of test suites – Test design is no different (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 3 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 4

Practical (in)Adequacy Criteria Analogy: Building Codes • Criteria that identify inadequacies in test suites. • Building codes are sets of design rules – Examples – Maximum span between beams in ceiling, floor, and – if the specification describes different treatment in two cases, but walls; acceptable materials; wiring insulation; ... the test suite does not check that the two cases are in fact treated differently, we may conclude that the test suite is inadequate to – Minimum standards, subject to judgment of building guard against faults in the program logic. inspector who interprets the code – If no test in the test suite executes a particular program • You wouldn’t buy a house just because it’s “up statement, the test suite is inadequate to guard against faults in that statement. to code” • If a test suite fails to satisfy some criterion, the obligation – It could be ugly, badly designed, inadequate for your that has not been satisfied may provide some useful needs information about improving the test suite. • But you might avoid a house because it isn’t • If a test suite satisfies all the obligations by all the – Building codes are inadequacy criteria, like practical criteria, we do not know definitively that it is an effective test “adequacy” criteria test suite, but we have some evidence of its thoroughness. (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 5 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 6 Where do test obligations Some useful terminology come from? • Test case : a set of inputs, execution conditions, and a • Functional (black box, specification-based): from pass/fail criterion. software specifications • Test case specification : a requirement to be satisfied by • Example: If spec requires robust recovery from power failure, test one or more test cases. obligations should include simulated power failure • Structural (white or glass box): from code • Test obligation : a partial test case specification, requiring some property deemed important to thorough • Example: Traverse each program loop one or more times. testing. • Model-based: from model of system • Test suite : a set of test cases. • Models used in specification or design, or derived from code • Test or test execution : the activity of executing test • Example: Exercise all transitions in communication protocol model cases and evaluating their results. • Fault-based: from hypothesized faults (common bugs) • Adequacy criterion : a predicate that is true (satisfied) • Example: Check for buffer overflow handling (common or false (not satisfied) of a � program, test suite � pair. vulnerability) by testing on very large inputs (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 7 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 8

Adequacy criteria Satisfiability • Adequacy criterion = set of test obligations • Sometimes no test suite can satisfy a criterion for a given program • A test suite satisfies an adequacy criterion if – Example: Defensive programming style includes – all the tests succeed (pass) “can’t happen” sanity checks – every test obligation in the criterion is satisfied by if (z < 0) { at least one of the test cases in the test suite. throw new LogicError( – Example: “z must be positive here!”) } the statement coverage adequacy criterion is satisfied by test suite S for program P if each executable statement in No test suite can satisfy statement coverage for this P is executed by at least one test case in S, and the program (if it’s correct) outcome of each test execution was “pass”. (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 9 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 10 Coping with Unsatisfiability Coverage: Useful or Harmful? • Measuring coverage (% of satisfied test • Approach A: exclude any unsatisfiable obligations) can be a useful indicator ... obligation from the criterion. – Of progress toward a thorough test suite, of trouble – Example: modify statement coverage to require spots requiring more attention execution only of statements that can be executed. • ... or a dangerous seduction – But we can’t know for sure which are executable! – Coverage is only a proxy for thoroughness or • Approach B: measure the extent to which a test adequacy suite approaches an adequacy criterion. – It’s easy to improve coverage without improving a test suite (much easier than designing good test – Example: if a test suite satisfies 85 of 100 cases) obligations, we have reached 85% coverage . – The only measure that really matters is • Terms: An adequacy criterion is satisfied or not, a coverage (cost-)effectiveness measure is the fraction of satisfied obligations (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 11 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 12

Comparing Criteria The subsumes relation Test adequacy criterion A subsumes test • Can we distinguish stronger from weaker adequacy criteria? adequacy criterion B iff, for every program P, every test suite satisfying A with respect to P • Empirical approach: Study the effectiveness of also satisfies B with respect to P. different approaches to testing in industrial practice • Example: – What we really care about, but ... – Depends on the setting; may not generalize from one Exercising all program branches (branch coverage) organization or project to another subsumes exercising all program statements • Analytical approach: Describe conditions under which • A common analytical comparison of closely one adequacy criterion is provably stronger than related criteria another – Useful for working from easier to harder levels of – Stronger = gives stronger guarantees coverage, but not a direct indication of quality – One piece of the overall “effectiveness” question (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 13 (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 14 Uses of Adequacy Criteria • Test selection approaches – Guidance in devising a thorough test suite • Example: A specification-based criterion may suggest test cases covering representative combinations of values • Revealing missing tests – Post hoc analysis: What might I have missed with this test suite? • Often in combination – Example: Design test suite from specifications, then use structural criterion (e.g., coverage of all branches) to highlight missed logic (c) 2007 Mauro Pezzè & Michal Young Ch 9, slide 15