Testing: Our Experiences Test Case Sof tware Testing Software to - - PDF document

1

Sof tware Testing Testing: Our Experiences

Software to be tested Test Case Output

When to Stop?

Software to be tested Test Case Output Enough? No Yes

Test Case Generation Verif ication Test Coverage

Sorting Program Sorting Program

A Real Testing Example

SPECS: Takes a list

• f numbers;

returns a sorted list. {1,3,2} {1,2,3} {3,2,3} {} {-1, -2} Just a list. A sorted list. Repeated entry. Empty list. Negative numbers. Test Cases {1, 2, 3} Output {1, 2, 3} Output {2, 3, 3} Output {} Output {-2, -1} Output

Automated Testing

Software to be tested Test Case Output Enough? No Yes

Test Case Generation Verif ication Test Coverage

Automated Testing

Software to be tested Test Case Output

Coverage Evaluator Test Case Generator Verif ier OR Test Oracle Test Specs

2

Testing the New Version

Original Test Cases Original Sof tware Modif ied Sof tware New Test Cases

Regression Testing

Original Test Cases Original Sof tware Modif ied Sof tware New Test Cases

• Process of determining whether a

task has been correctly carried out [Schach ’96]

• Goals of testing

– Reveal Fault s

• Correct ness
• Reliabilit y
• Usabilit y
• Robust ness
• Perf ormance

What is Testing?

Conf licting Goals?

Types of Testing

• Execution- based Testing
• Non- execution based Test ing
• Discussion

Execution- based Testing

• Generating and Executing Test

Cases on the Sof tware

• Types of Execution- based Testing

– Testing to Specif ications

• Black-box Test ing

– Testing to Code

• Glass-box (Whit e-box) Test ing

Black- box Testing

• Discussion: MAC/ ATM Machine

Example

– Specs

• Cannot wit hdraw more t han $300
• Cannot wit hdraw more t han your account

balance

Sof tware x Balance

3

White- box Testing

• Example

I NPUT-FROM-USER(x); I f (x < = 300) { I NPUT-FROM-FI LE(BALANCE); I f (x < = BALANCE) GiveMoney x; else Print “You don’t have $x in your account !!”} else Pr int “You cannot wit hdr aw mor e t han $300”; Ej ect Car d; 1 2 3 4 5 6 x: 1. . 1000; Generate test cases to cover each statement

Discussion

• Which is superior?
• Each technique has its strengths –

Use both

Determining Adequacy

• Statement coverage
• Branch coverage
• Path coverage
• All- def - use- path coverage

Surprise Quiz

• Determine test cases so t hat each

print statement is executed at least once

input (x); if (x < 100) pr int “Line 1”; else { if (x < 50) pr int “Line 2” else pr int “Line 3”; } if 1 if 2 3 end begin

x<100 x>=100 x>=50 x<50

x>=100 x>=100

Non- execution Based

• Walkthroughs

– Manual simulation by team leader

• I nspections

– Developer narrates the reading

• Key I dea

– Review by a team of experts: Syntax checker?

• Code Readings
• Formal Verif ication of Correct ness

– Very Expensive – Justif ied in Critical Applications

• Semi- f ormal: Some Assert ions

Simulation

• I ntegration wit h system hardware is

central to t he design

• Model the external hardware
• Model the interf ace
• Examples
• Discussion

4

Boundary- value Analysis

• Partition the program domain into

input classes

• Choose test data that lies both

inside each input class and at the boundary of each class

• Select input that causes output at

each class boundary and within each class

• Also known as st ress testing

Testing Approaches

• Top- down
• Bottom- up
• Big Bang
• Unit testing
• I ntegration testing
• Stubs
• System testing

Mutation Testing

• Errors are introduced in the

program to produce “mutants”

• Run test suite on all mutants and

the original program

Test Case Generation

• Test I nput to the Sof tware
• Some researchers/ authors also

def ine the test case to contain the expected output f or the test input

Category- partition Method

• Key idea

– Method f or creating f unctional test suit es – Role of test engineer

• Analyze t he syst em specif icat ion
• Writ e a series of f ormal t est specif icat ions

– Automat ic generator

• Produces t est descript ions

Steps

• Decompose t he f unctional specif ication

into f unctional units

– Characteristics of f unctional units

• They can be t est ed independent ly
• Examples

– A t op-level user command – Or a f unct ion

• Decomposition may require several stages
• Similar to high- level decomposition done

by sof tware designers

– May be reused, although independent decomposition is recommended

5

Steps

• Examine each f unctional unit

– I dentif y parameters

• Explicit input t o t he f unct ional unit

– Environmental condit ions

• Charact erist ics of t he syst em’s st at e
• Test Cases

– Specif ic values of parameters – And environmental conditions

Steps

• “Test cases are chosen to maximize

chances of f inding errors”

• For each parameter & environmental

condition

– Find categories

• Maj or proper t y or char act er ist ic
• Examples

– Br owser s, Operat ing Syst ems, arr ay size

• For each cat egor y

– Find choices » Examples: (I E 5.0, I E 4.5, Net scape 7.0), (Windows NT, Linux), (100, 0, -1)

Steps

• Develop “Formal Test Specif ication”

f or each f unctional unit

– List of categories – Lists of choices within each category

• Constraints
• Automatically produces a set of

“test f rames”

– Consists of a set of choices

AI Planning Method

• Key I dea

– I nput to Command- driven sof t ware is a sequence of commands – The sequence is like a plan

• Scenario to test

– I nitial state – Goal state

Example

• VCR command- line sof tware
• Commands

– Rewind

• I f at t he end of t ape

– Play

• I f f ully rewound

– Eject

• I f at t he end of t ape

– Load

• I f VCR has no t ape

Preconditions & Ef f ects

• Rewind

– Precondition: I f at end of tape – Ef f ects: At beginning of tape

• Play

– Precondition: I f at beginning of tape – Ef f ects: At end of tape

• Eject

– Precondition: I f at end of tape – Ef f ects: VCR has no tape

• Load

– Precondition: I f VCR has no tape – Ef f ects: VCR has tape

6

Preconditions & Ef f ects

• Rewind

– Precondition: end_of _tape – Ef f ects: ¬ ¬ ¬ ¬end_of _tape

• Play

– Precondition: ¬ ¬ ¬ ¬end_of _tape – Ef f ects: end_of _tape

• Eject

– Precondition: end_of _tape – Ef f ects: ¬ ¬ ¬ ¬has_tape

• Load

– Precondition: ¬ ¬ ¬ ¬has_tape – Ef f ects: has_tape

I nitial and Goal States

• I nitial State

– end_of _tape

• Goal St ate

– ¬ ¬ ¬ ¬end_of _tape

• Plan?

– Rewind

I nitial and Goal States

• I nitial State

– ¬ ¬ ¬ ¬end_of _tape & has_tape

• Goal St ate

– ¬ ¬ ¬ ¬has_tape

• Plan?

– Play – Eject

Test Coverage & Adequacy

• How much testing is enough?
• When to stop testing
• Test data selection criteria
• Test data adequacy criteria

– Stopping rule – Degree of adequacy

• Test coverage criteria
• Objective measurement of test

quality

Preliminaries

• Test data selection

– What test cases

• Test data adequacy criteria

– When to stop testing

• Examples

– Statement Coverage – Branch coverage – Def - use coverage – Path coverage

Goodenough & Gerhart [‘75]

• What is a sof tware test adequacy

criterion

– Predicate that def ines “what properties of a program must be exercised to const itute a thorough test ”, i. e. , one whose successf ul execution implies no errors in a tested program

7

Uses of test adequacy

• Objectives of testing
• I n terms that can be measured

– For example branch coverage

• Two levels of testing

– First as a stopping rule – Then as a guideline f or additional test cases

Categories of Criteria

• Specif ication based

– All- combination criterion

• choices

– Each- choice- used criterion

• Program based

– Statement – Branch

• Note t hat in bot h the above t ypes, the

correctness of the out put must be checked against the specif icat ions

Others

• Random testing
• Statistical testing
• I nterf ace based