{bnielsen@cs.aau.dk} } Testing Concepts Brian Nielsen {
Com plex System s
CISS A very com plex system t l A
Spectacular softw are bugs p g Ariane 5 The first Ariane 5 rocket was launched in June, 1996. It used software developed for the successful Ariane 4. The rocket successful Ariane 4 The rocket carried two computers, providing a backup in case one computer failed during launch. Forty seconds into its maiden flight the seconds into its maiden flight, the rocket veered off course and exploded. The rocket, along with Ariane 5 was a much more $500 million worth of satellites, powerful rocket and generated was destroyed. d t d forces that were larger than the computer could handle. Shortly after launch, it received an input value that was too large The value that was too large. The main and backup computers shut down, causing the rocket to veer off course. CISS
Spectacular softw are bugs p g U.S.S. Yorktow n, U.S. Navy When the sailor entered the In 1998, the USS Yorktown mistaken number, the computer became the first ship to test the tried to divide by zero, which isn't US Navy's Smart Ship program. possible. The software didn't The Navy planned to use off-the- The Navy planned to use off the check to see if the inputs were check to see if the inputs were shelf computers and software valid before computing and generated an invalid answer that instead of expensive U.S.S. was used by another computer. Yorktown, courtesy of U.S. Navy The error cascaded several The error cascaded several custom made machines A sailor custom-made machines. A sailor computers and eventually shut mistakenly entered a zero for a down the ship's engines. data value on a computer. Within minutes, Yorktown was dead in th the water. It was several hours t It l h before the ship could move again. CISS
Spectacular softw are bugs p g Moon or Missiles The United States established the Ballistic Missile Early Warning B lli ti Mi il E l W i System (BMEWS) during the Cold War to detect a Soviet missile attack. On October 5, 1960 the BMEWS radar at Thule, The radar had actually detected Greenland detected something. the Moon rising over the horizon. Its computer control system Unfortunately, the BMEWS decided the signal was made by g y computer had not been t h d t b hundreds of missiles coming programmed to understand what toward the US. the moon looked like as it rose in the eastern sky, so it interpreted the huge signal as Soviet missiles. Luckily for all of us, the mistake was realized in time. CISS
Spectacular softw are bugs p g Therac 2 5 The Therac-25 was withdrawn from use after it was determined that it could deliver fatal h i ld d li f l overdoses under certain The Therac-25 radiation therapy conditions. The software would machine was a medical device shut down the machine before shut down the machine before that used beams of electrons or h d b f l delivering an overdose, but the photons to kill cancer cells. error messages it displayed were Between 1985-1987, at least six so unhelpful that operators people got very sick after Therac- couldn't tell what the error was, ld 't t ll h t th 25 treatments. Four of them died. The manufacturer was or how serious it was. In some confident that their software cases, operators ignored the made it impossible for the p message completely. message completely. machine to harm patients. IEEE Computer IEEE Computer , Vol. 26, No. 7, July 1993, pp. 18 IEEE Computer IEEE Computer , Vol. 26, No. 7, July 1993, pp. 18 , Vol. 26, No. 7, July 1993, pp. 18- , Vol. 26, No. 7, July 1993, pp. 18- -41 -41 41 41 CISS
Spectacular Softw are Bugs p g …. continued INTEL Pentium II floating-point division 470 Mill US $ 470 Mill US $ Baggage handling system, Denver 1.1 Mill US $/ day for 9 months ill S $/ d f 9 h Mars Pathfinder … … . CISS
O di Ordinary Softw are Bugs S ft B BMW 745i software Defect: "On certain passenger vehicles, due to a software error, a desynchronization of the valvetronic m otors for engine banks I and II may occur If this m otors for engine banks I and II may occur. If this occurs, the engine could stall. In those cases, the driver may not be able to restart the engine. Depending on the level of engine roughness, or stalling, as well as traffic conditions and the driver’s reactions, this could lead to a crash." 15000 recalled 70-100 ECU’s in modern cars SW major part of development cost CISS
Com ponent-Based Devel. “A component is a reusable unit of com position with explicitly specified, provided, and required interfaces and quality attributes that denotes a single abstraction and quality attributes, that denotes a single abstraction and can be composed without modifications”[ Gross’05] How to check that a component or a component-based system has sufficient quality? Today’s component specs are weak (list of public methods) exceptionally pre- and post-conditions • CISS Behavior not specified. •
Com ponent Testing Stakeholders Provider Users Challenges Testing the component in Testing the component in a new context Lack of access to the internal workings of a internal workings of a component (observability and controllability) Adequate component Adequate component testing - adaptation testing CISS
Testing Testing Testing: Testing: to check the quality (functionality, reliability, performance, … ) of an (software) object -by performing experiments -in a controlled way • In avg 10-20 errors per 1000 LOC • In avg. 10 20 errors per 1000 LOC • 30-50 % of development time and cost in embedded software To find errors To determine risk of release CISS
T Testing ti Dynam ic testing is the process of executing a i th f ti D i t ti program or system with the intent of finding error (Glenford Meyers’ definition) Static testing is any activity that aims at finding defects by inspecting, reviewing, walking fi di d f t b i ti i i lki through, and analyzing any static component of the software (code, documents, and models) o a ( od , do u , a d od ) Debugging is an ad hoc activity performed by individual developers to find and remove bugs f from a program. Testing is a planned activity CISS
W hat is a Test? W hat is a Test? Test Cases Test Cases Output Test Data Correct result? Software under Test under Test O Oracle l CISS
Testing process g p T requirements Planning& Control L preparation test plan specification I O L : Life-cycle for testing abstract/logical Analysis O : Organization test cases and test I : Infrastructure and tools specification design g T : Techniques q concrete/ Test executable implemen- test cases tation Test verdict/logs implementation execution Evaluation quality report and reporting reporting CISS
T Types of Testing f T ti Level system integration unit black box Accessibility white box efficiency efficiency usability reliability reliability functionality Aspect CISS
Quality-Characteristics (ISO-9126) Quality-Characteristics (ISO-9126) functional testing Functionality Suitability, accuracy, security, compliance, interoperability y, y, y, p , p y reliability testing Reliability maturity, fault tolerance, recoverability usability testing Usability understandability, learnability, operability performance testing performance testing Efficiency Efficiency time behaviour, resource utilization maintainability testing ?? maintainability testing ?? Maintainability y Analysability, changeability, stability, testability portability testing ? Portability Adaptability, installability, conformance, replaceability CISS
W hitebox Testing W hitebox Testing if int invoice (int x, int y) { ( , y) { s>1000 1000 x>30 if int d1, d2, s; d1=95 if (x<=30) d2=100; d2=100 d2=90 else d2=90; d1=80 d1 80 s=5*x + 10 *y; if (s<=200) d1=100; else if (s<=1000) d1 = 95; else d1 = 80; s>200 if return (s*d1*d2/10000); return } d1=100 Test Cases Test Data Expected Output X=5 Y=5 75 X=31 Y=10 229.5 X=30 Y=100 977.5 CISS
CISS x x domain testing Blackbox testing Blackbox testing output y SUT events requirements input input
V - Model user acceptance acceptance test spec test requirements system system test spec specification p test component architecture integration test spec integration spec test component/ module test spec module detailed design detailed design t test t unit test spec test spec implementation unit-test code CISS
CISS Module/ Com ponent Test test stub Component under p test driver e test test d / test stub
CISS M12 M12 M9 M4 M8 t I ntegration Test M11 M11 T M1 M1 M3 M7 ti Design Hierarchy Design Hierarchy M10 M10 M2 M6 t ”consists-of” ”calls/uses” M5 I
I I ntegration Test t ti T t Module Hierarchy Module Hierarchy M1 M1 Top-Down T D Buttom-Up B tt U ”calls/uses” Integration Integration ”consists-of” M2 M3 M4 M9 M7 M8 M5 M6 Sandwich Integration M10 M10 M11 M11 M12 M12 Depth-first vs. bredth first CISS
Recommend
More recommend
