SE320 Software Verification and Validation Requirements Testing, Code Testing Frameworks Prof. Colin S. Gordon Fall 2018, Week 2 (10/01–05) 1
Prof. Gordon’s Office Hours Tuesdays 2-4pm 2
The Problem of Requirements: An Example 3
The Triangle Program • Assume that I have been asked to write a program satisfying the following requirements: The program will accept as input three integers: a, b, and c. These integers represent the sides of a triangle. The program will print out the type of triangle as deter- mined by the three sides: Equilateral (i.e., all 3 sides are equal), Isosceles (i.e., 2 sides are equal), Scalene (i.e., the 3 sides are all different), or NotATriangle. • I have written the Triangle program, and I’d like you to test it. • How would you verify this program? 4
Possible Test Cases id (a, b, c) Expected Outcome T1 5, 5, 5 Equilateral T2 2, 2, 3 Isosceles T3 3, 4, 5 Scalene T4 3, 4, 20 NotATriangle Question Any other test cases? • Other values for Equilateral (e.g., (1,1,1) or (100,100,100))? • What about Isosceles with b=c (e.g., (3,7,7)) and a=c (e.g., (10,5,10))? • What about Scalene with a > b > c (e.g., (5,4,3))? • What about (5,1,1)? 5
Possible Test Cases • What about negative values? • What should the expected output be for negative values? • What should we expect for (-10,-10,-10)? Equilateral, or something else? 6
Questions and Observations • Can we claim that we have “verified” the program by using test cases {T1, T2, T3, T4}? • At what point are we satisfied that we have adequately tested the program? • Notice there are infinitely many possible input values • Okay, 3 ∗ 2 | bits in integer | , which is only 12,884,901,888 for 32-bit integers — roughly 12.9 billion inputs 7
Questions and Observations (cont.) The requirements are quite weak. • What happens with non-positive inputs? • Should (0,5,5) be considered Isosceles? • If not, what is it? • What about (0,0,0)? • What if the user provides only 2 input values? • Might depend on the language, but easy in JavaScript/etc., command line applications • There is no explicit upper limit on the input values • Is a 32-bit integer big enough? • There is no clear specification of what constitutes “NotATriangle” • What assumptions did we make for “NotATriangle”? 8
Improved Requirements The program will accept as input three integers: a, b, and c. These integers represent the sides of a triangle. The integers a, b, and c must satisfy the following con- ditions: C1 1 ≤ a ≤ 200 C2 1 ≤ b ≤ 200 C3 1 ≤ c ≤ 200 C4 a < b + c C5 b < a + c C6 c < a + b The output of the program is the type of triangle deter- mined by the three sides: Equilateral, Isosceles, Sca- lene, or NotATriangle. 9
Improved Requirements (cont.) If an input value fails any of conditions C1 C2 and C3, the program notes this with a message, for example, "Value of b is not in the range of permitted values". If values of a, b, and c satisfy conditions C1, C2, and C3, one of the four mutually exclusive outputs is given: 1. If all three sides are equal, the program output is Equilateral. 2. If exactly one pair of sides is equal, the program outputs Isosceles. 3. If no pair of sides is equal, the program output is Scalene. 4. If any of conditions C4, C5, and C6 fails, the program output is NotATriangle. 10
Observations Obviously this version of the requirements is much better. • There is a clear definition of the acceptable range of input values • The developer does not have to make wild assumptions about NotATriangle • The tester has more information to work with 11
Observations (cont.) • The quality of the requirements matters a great deal even for very simple programs! • It matters even more for complex software systems!!! • How do we know that the requirements are good? • Even with good requirements we still have a testing problem. . . • How do we know something is still not missing or is incorrectly specified • What if the program implements some function which is not even specified in the requirements? • For example, attempts to determine a “right triangle”. . . 12
Important Questions • Two important technical questions related to verification: 1. How do we go about generating test cases? 2. What criteria do we use to ensure that we have adequately tested the program? • One important process question: 1. What is the recommended process? 13
Specification-based vs. Program-based Testing • There are two major testing approaches: • Specification-based Testing • Program-based Testing • Specification-based testing uses the requirements as the point of reference for generating test cases and for determining adequacy of testing. • i.e., Blackbox Testing • Program-based testing uses the code as the point of reference. • i.e., Whitebox Testing • As we’ll see, each approach has its advantages and disadvantages. • As you can imagine, it is possible to combine approaches – we’ll see how. 14
Specification Testing 15
“Testing” the Specification 1 • You do not need code in order to start testing • “Testing” the specification can save time and cost later on • What mistakes would you have made in the Triangle program example, with the old vs. new spec? • Mistakes in the specifications account for half of all bugs • The specification is typically written using prose and pictures to describe functional and non-functional aspects of the software 1 Based on material from Software Testing, 2nd Ed., by Ron Patton 16
Motivation • Consider a spec that is vague, inconsistent, incomplete, or otherwise inadequate for describing a useful software system • Any software “correctly” implementing this broken specification is inherently broken • Therefore finding problems with the specification effectively rules out certain bugs in the implementation • Or at least, greatly reduces their odds and clarifies how to fix 17
Requirements Specification: An Overview • Basic goal: To understand the problem as perceived by the user. • Activities of specification are problem oriented. • Focus on what, not how (this is design) • Don’t cloud the specification with unnecessary detail. • Don’t pre-constrain design in the specification. • After specification is done, do software design: • solution oriented • how to implement the what • Key to specification is good communication between customer and developers. • Work from specification document as guide. 18
Requirements Specification • Basically, it’s a process of setting clear and precise expectations of the customer about the software 19
The Purpose of Specification • Raw user requirements are often: • vague • contradictory • impractical or impossible to implement • overly concrete • just plain wrong • The purpose of specification is to get a usable set of requirements from which the system may be designed and implemented, with minimal “surprises”. 20
Two Kinds of Requirements • Functional (what): The precise tasks or functions the system is to perform. • e.g., details of a flight reservation system • Non-functional (how): Usually, a constraint of some kind on the system or its construction • e.g., expected performance and memory requirements, process model used, implementation language and platform, compatibility with other tools, deadlines, . . . 21
The Specification Document • The official statement of what is required of the system developers. • Includes system models, requirements definition, and requirements specification. • Not a design document. • States functional and non-functional requirements. • Serves as a reference document for maintenance. 22
Specification Document “Requirements” • Should be easy to change as requirements evolve • Must be kept up-to-date as system changes 23
Specification Should State. . . • Forseen problems • “won’t support Windows 7” • Expected evolution • “will port to Linux and FreeBSD in next version” • Response to unexpected events/usage • “if input data is in the old format, will auto-convert” 24
Example Specification Structure • Introduction (describe need for system) • Functional Requirements • Non-Functional Requirements • System Evolution (describe anticipated changes) • Glossary (technical and/or new jargon) • Appendices • Index Why a glossary? 25
To summarize. . . • Specification focuses on determining what the customer wants, and not how it will be implemented. • Specification is hard to get correct; it requires good communication skills. • Requirements may change over time. • Requirements specification requires iteration. • The customer often doesn’t have good grasp of what he wants. • Bugs created in the requirements stage are very expensive to fix later. 26
Recommend
More recommend
