Testing: Our Experiences Test Case Sof tware Testing Software to - PDF document

Testing: Our Experiences Test Case Sof tware Testing Software to be tested Output When to Stop? A Real Testing Example Test Case Generation Test Cases Just a list. Test Case {1,3,2} SPECS: A sorted list. {1,2,3} Takes a list Repeated entry. of numbers; {3,2,3} Software Empty list. returns a to be tested {} Sorting Sorting Negative numbers. sorted list. Program {-1, -2} Verif ication Program Output {1, 2, 3} {2, 3, 3} {1, 2, 3} {-2, -1} {} Output Output Output Output Output No Test Coverage Enough? Yes Automated Testing Automated Testing Test Case Generation Test Case Test Case Test Case Generator Software Software to be tested to be tested Verif ication Test Specs Verif ier Output Output OR Test Oracle No Test Coverage Coverage Enough? Evaluator Yes 1

Testing the New Version Regression Testing Original Original Original Original Test Test Sof tware Sof tware Cases Cases New New Modif ied Modif ied Test Test Sof tware Sof tware Cases Cases What is Testing? Types of Testing • Process of determining whether a • Execution- based Testing task has been correctly carried out • Non- execution based Test ing [Schach ’96] • Goals of testing – Reveal Fault s • Correct ness • Reliabilit y • Usabilit y Conf licting Goals? • Robust ness • Discussion • Perf ormance Execution- based Testing Black- box Testing • Generating and Executing Test • Discussion: MAC/ ATM Machine Cases on the Sof tware Example – Specs • Types of Execution- based Testing • Cannot wit hdraw more t han $300 – Testing to Specif ications • Cannot wit hdraw more t han your account • Black-box Test ing balance – Testing to Code Balance • Glass-box (Whit e-box) Test ing x Sof tware 2

White- box Testing Discussion • Example • Which is superior? Generate test cases • Each technique has its strengths – x: 1. . 1000; to cover each statement Use both 1 I NPUT-FROM-USER(x); I f (x < = 300) { 2 I NPUT-FROM-FI LE(BALANCE); I f (x < = BALANCE) 3 GiveMoney x; 4 else Print “You don’t have $x in your account !!”} else 5 Pr int “You cannot wit hdr aw mor e t han $300”; 6 Ej ect Car d; Determining Adequacy Surprise Quiz • Determine test cases so t hat each • Statement coverage print statement is executed at least once • Branch coverage begin • Path coverage input (x); • All- def - use- path coverage if if (x < 100) x>=100 x<100 pr int “Line 1”; 1 if x>=100 else { x>=100 x<50 x>=50 if (x < 50) pr int “Line 2” else pr int “Line 3”; 2 3 } end Non- execution Based Simulation • Walkthroughs • I ntegration wit h system hardware is – Manual simulation by team leader central to t he design • I nspections • Model the external hardware – Developer narrates the reading • Model the interf ace • Key I dea – Review by a team of experts: Syntax checker? • Examples • Code Readings • Formal Verif ication of Correct ness • Discussion – Very Expensive – Justif ied in Critical Applications • Semi- f ormal: Some Assert ions 3

Boundary- value Analysis Testing Approaches • Partition the program domain into • Top- down input classes • Bottom- up • Choose test data that lies both • Big Bang inside each input class and at the boundary of each class • Unit testing • Select input that causes output at • I ntegration testing each class boundary and within each • Stubs class • System testing • Also known as st ress testing Mutation Testing Test Case Generation • Errors are introduced in the • Test I nput to the Sof tware program to produce “mutants” • Some researchers/ authors also • Run test suite on all mutants and def ine the test case to contain the the original program expected output f or the test input Category- partition Method Steps • Decompose t he f unctional specif ication • Key idea into f unctional units – Method f or creating f unctional test – Characteristics of f unctional units suit es • They can be t est ed independent ly – Role of test engineer • Examples – A t op-level user command • Analyze t he syst em specif icat ion – Or a f unct ion • Writ e a series of f ormal t est specif icat ions • Decomposition may require several stages – Automat ic generator • Similar to high- level decomposition done • Produces t est descript ions by sof tware designers – May be reused, although independent decomposition is recommended 4

Steps Steps • “Test cases are chosen to maximize • Examine each f unctional unit chances of f inding errors” – I dentif y parameters • For each parameter & environmental • Explicit input t o t he f unct ional unit condition – Environmental condit ions – Find categories • Charact erist ics of t he syst em’s st at e • Maj or proper t y or char act er ist ic • Test Cases • Examples – Br owser s, Operat ing Syst ems, arr ay size – Specif ic values of parameters • For each cat egor y – And environmental conditions – Find choices » Examples: (I E 5.0, I E 4.5, Net scape 7.0), (Windows NT, Linux), (100, 0, -1) Steps AI Planning Method • Develop “Formal Test Specif ication” • Key I dea f or each f unctional unit – I nput to Command- driven sof t ware is a sequence of commands – List of categories – The sequence is like a plan – Lists of choices within each category • Scenario to test • Constraints – I nitial state • Automatically produces a set of – Goal state “test f rames” – Consists of a set of choices Example Preconditions & Ef f ects • VCR command- line sof tware • Rewind – Precondition: I f at end of tape • Commands – Ef f ects: At beginning of tape – Rewind • Play • I f at t he end of t ape – Precondition: I f at beginning of tape – Play – Ef f ects: At end of tape • I f f ully rewound • Eject – Eject – Precondition: I f at end of tape • I f at t he end of t ape – Ef f ects: VCR has no tape – Load • Load • I f VCR has no t ape – Precondition: I f VCR has no tape – Ef f ects: VCR has tape 5

Preconditions & Ef f ects I nitial and Goal States • Rewind • I nitial State – Precondition: end_of _tape – end_of _tape – Ef f ects: ¬ ¬ ¬ ¬ end_of _tape • Goal St ate • Play – ¬ ¬ end_of _tape ¬ ¬ – Precondition: ¬ ¬ ¬ ¬ end_of _tape – Ef f ects: end_of _tape • Plan? • Eject – Rewind – Precondition: end_of _tape – Ef f ects: ¬ ¬ has_tape ¬ ¬ • Load – Precondition: ¬ ¬ has_tape ¬ ¬ – Ef f ects: has_tape I nitial and Goal States Test Coverage & Adequacy • How much testing is enough? • I nitial State – ¬ ¬ ¬ end_of _tape & has_tape ¬ • When to stop testing • Test data selection criteria • Goal St ate – ¬ ¬ ¬ ¬ has_tape • Test data adequacy criteria – Stopping rule • Plan? – Degree of adequacy – Play • Test coverage criteria – Eject • Objective measurement of test quality Preliminaries Goodenough & Gerhart [‘75] • Test data selection • What is a sof tware test adequacy – What test cases criterion • Test data adequacy criteria – Predicate that def ines “what – When to stop testing properties of a program must be exercised to const itute a thorough • Examples test ”, i. e. , one whose successf ul – Statement Coverage execution implies no errors in a tested – Branch coverage program – Def - use coverage – Path coverage 6

Uses of test adequacy Categories of Criteria • Specif ication based • Objectives of testing – All- combination criterion • I n terms that can be measured • choices – For example branch coverage – Each- choice- used criterion • Two levels of testing • Program based – Statement – First as a stopping rule – Branch – Then as a guideline f or additional test • Note t hat in bot h the above t ypes, the cases correctness of the out put must be checked against the specif icat ions Others • Random testing • Statistical testing • I nterf ace based 7