Test Code Coverage White box testing Statement Coverage example 1 - PowerPoint PPT Presentation

Test Code Coverage White box testing

Statement Coverage – example 1 n Make sure every statement in the program is executed at least once T F n Example 1 2 n if a < b then c = a+b ; d = a*b /* 1 */ 
 else c = a*b ; d = a+b /* 2 */ 
 F T if c < d then x = a+c ; y = b+d /* 3 */ 
 else x = a*c ; y = b*d /* 4 */ 4 3 n Statement coverage – can do with 2 tests n Execute 1 & 3 with a < b & a+b < a*b n a = 2 ; b = 5 n Execute 2 & 4 with a >= b & a*b >= a+b n a = 5 ; b = 2 Test–2

Statement Coverage n Loops – only 1 test required n Execute body at least once n If there are choices in the body, need to execute body with enough tests to execute the statements in the body, analogous to example 1. F T Body Test–3

Statement Coverage How n How do you know you have statement coverage? TCC–4

Statement Coverage How n How do you know you have statement coverage? n Instrument your program with an array of flags initialized to false n Set a unique flag to true in each block of statements n Run your test n If all flags are true then you have achieved statement coverage TCC–5

Problems with statement coverage n A statement must be executed with different values for the relevant variables to be fully tested n Loop bodies may need to be iterated many times to reveal issues n Not all statements are equally important n Only the true branch of an if statement may be executed but coverage may be 90% for the statement if the false branch is one tenth of the size TCC–6

Next level of improvement n Path coverage n Alias 1: Segment coverage n Alias 2: DD coverage n Decision to decision coverage TCC–7

Path Coverage – example 1 n Every path in the program is executed at least once T F n Example 1 2 n if a < b then c = a+b ; d = a*b /* 1 */ 
 else c = a*b ; d = a+b /* 2 */ 
 F T if c < d then x = a+c ; y = b+d /* 3 */ 
 else x = a*c ; y = b*d /* 4 */ 4 3 n Path coverage – 4 tests n Execute 1 & 3 with a < b & a+b < a*b 
 a = 2 ; b = 5 n Execute 2 & 4 with a >= b & a*b >= a+b a = 5 ; b = 2 Add for path n Execute 1 & 4 with a < b & a+b >= a*b a = 0 ; b = 1 n Execute 2 & 3 with a >= b & a*b < a+b a = 1 ; b = 0 TCC–8

Path Coverage – example 1 n Loops – 3 tests required n Execute body zero times, once in the path, many in the path n Once is not enough as frequently first time through is a special case 
 n Path coverage usually requires exponential increase in tests as the number of choices and loops increases n Due to multiplication n two loops in sequence – 9 tests n three loops in sequence – 27 tests n ten if....then...else in sequence – 1024 tests TCC–9

Path Coverage – example 2 – the problem n Convert an integer represented as a decimal string to a real number. n ASCII string "123.456" ==> 123.456 in binary n The EBNF for the input Input ::= +[ Spaces ] [ + , - ] [ IntegerPart ] [ '.' [ DecimalPart ] ]; IntegerPart ::= +( DecimalDigit ); DecimalPart ::= +( DecimalDigit ); DecimalDigit ::= ( '0' , '1', '2' ,'3' ,'4' ,'5' ,'6' ,'7' ,'8' , '9'); TCC–10

Path Coverage – example 2 – the algorithm 1 Skip any leading spaces 2 Determine what the sign of the number is 3 Get the integer part of the number by scanning either to the end of the number or a decimal point 4 Continue building the integer representation of the input as if there was no decimal point, meanwhile counting the number of decimal digits 5 Compute the real number representation from the sign, integer representation and count of decimal digits TCC–11

Path Coverage – example 2 – test cases n 2 tests are sufficient for statement coverage n positive and negative real numbers. n 162 tests estimated for all paths. n 3 cases first loop – step 1 skip lead spaces n 3 cases first if statement – step 2 determine sign n 3 cases second loop – step 3 get integer part n 2 cases second if statement – step 3 check decimal point n 3 cases third loop – step 4 get decimal part 
 n Not all cases are possible -- for example if there is no '.' (second if statement), then the third loop cannot not be executed one or many times, only zero times. TCC–12

Path Coverage How n How to you know you have path coverage? TCC–13

Path Coverage How n How to you know you have path coverage? n As for statement coverage have an array of flags with a flag in each block of statements n Compare the pattern of true flags with the expected statement blocks in each path n Continue until every path pattern has been matched TCC–14

Other kinds of coverage n Branch coverage n Every edge from a statement node is executed at least once n At least one true and one false evaluation for each predicate 
 TCC–15

Other kinds of coverage n Multi-condition coverage n All true-false combinations of simple conditions in compound predicates are considered at least once n Guarantees statement, branch and predicate coverage n Does not guarantee path coverage TCC–16

Other kinds of coverage n Dataflow coverage n Focuses on the points at which variables receive values and the points at which these values are used n And more … 
 EECS 4313 – Software Engineering Testing n TCC–17

When is testing done? TCC–18

When is testing done? n Short answer: Never! TCC–19

When is testing done? n Short answer: Never! n A bit longer answer: n When all features of the system have been tested with all possible inputs that could make a difference TCC–20

When is testing done? n Short answer: Never! n A bit longer answer: n When all features of the system have been tested with all possible inputs that could make a difference n In practice this is hard to determine TCC–21

When is testing done? n Short answer: Never! n A bit longer answer: n When all features of the system have been tested with all possible inputs that could make a difference n In practice this is hard to determine n Metrics such as code coverage can be used to give an idea of how sufficient the testing is TCC–22

Statement Code Metric n Observe the system as it is running n Keep track of how many of the statements in the code were executed at least once n Divide by the total number of statements in the system n A comprehensive test suite is important n Typically, it is hard to get high coverage n Anything above 70% is pretty good for a large system TCC–23

Software engineering guideline n Low code coverage indicates that more testing must be done n High code coverage gives little information about the quality of the testing n Let’s see a demo (EclEmma). TCC–24

Homework n Short term (days) n Install EclEmma and determine your statement test coverage 
 n Demonstrate your ability to do statement test coverage in Lab 9 
 n Long term (project) n Improve your testing to raise coverage close to100% 
 n Your final submission must discuss test code coverage in your Testing document TCC–25

EclEmma n Install EclEmma by doing the following in Eclipse n Select Help à Install New Software n In the install dialog enter the following in the Work with field, and press Return n http://update.eclemma.org/ n Check the latest EclEmma version and press Next n Follow the steps in the installation wizard 
 n Get the User Guide at the following location n http://eclemma.org/installation.html