CSE 331 Software Design and Implementation Lecture 8 Testing Zach Tatlock / Spring 2018
Outline • Why correct software matters – Motivates testing and more than testing, but now seems like a fine time for the discussion • Testing principles and strategies – Purpose of testing – Kinds of testing – Heuristics for good test suites – Black-box testing – Clear-box testing and coverage metrics – Regression testing
Non-outline • Modern development ecosystems have much built-in support for testing – Unit-testing frameworks like JUnit – Regression-testing frameworks connected to builds and version control – Continuous testing – … • No tool details covered here – See homework, section, internships, …
Clinical Neutron Therapy System
CNTS Beamline NN 141C FC3A Vault ROTATING SBD 5 ELEMENT PUMP Beam Monitor GROUP QUADRUPOLE BPM LENS 3 A2 Q3A 5 LENS 2 SBD 4 STEERING MAGNET LENS 1 X3A Focusing Y3A BPM 4 BEAM PLUG A PUMP STEERING Magnets GROUP MAGNET QUADRUPOLE A1 X2A LENS 3 FC2A Q2A Y2A LENS 2 A2 LENS 1 VALVE BPM 3 SBD 3 BPM 2A BEAM QUADRUPOLE (not in use) PROFILE SWITCHING Q1 A1 VALVE MONITOR MAGNET BPM STEERING STEERING LENS 1 LENS 2 LENS 3 1 PUMP PUMP PUMP BEAM MAGNET 2 MAGNET GROUP GROUP EXIT GROUP BLV X0 X1 Y1 SWM 21 ˚ Z-TGT VALVE Z1 2Z TARGET NN 143G ISO Beam Room SBD 3Z SBD 1 SBD 2 21 ˚ FARADAY 48 ˚ FARADAY CUP 1 CUP 2Z BPM 2B QUADRUPOLE B1 (not in use) VALVE LENS 1 BPM Q2B LENS 2 7 B Beam Plug NN 143J E M N D LENS 3 A I G N FIX Beam Room Faraday Cup G Cyclotron N SBD 3 E T FC2B VALVE C1 B2 VALVE GROUP PUMP BEAM PLUG B C SBD 3 BPM 6 STEERING EXPERIMENTAL MAGNET STATION QUADRUPOLE PUMP XG GROUP LENS 1 YG Q3B FC2C B1 LENS 2 LENS 3 EXPERIMENTAL VALVE C2 PUMP STATION SBD 4 GROUP B2 Vacuum Pump TARGET BPM 8 SBD 5 UNIVERSITY OF WASHINGTON MEDICAL CENTER CNTS - SEATTLE CLINICAL CYCLOTRON DATE: 18JAN12 DRAWING NUMBER: SEA 8001 REVISED: DRAWN BY: SS BEAMLINE LAYOUT TITLE:
CNTS Control Hardwired Safety Interlock System NN 141C A Vault F C 3 P S B D 5 O R E T A I T N T G N O E L G R U P P U O M L E M E Beamline Q R A D U A U 3 Q P S E L N 3 2 A P 5 M B 1 L S N E 2 S B D 4 S T E E R G E I T N L S E N G A X M N 3 A Therapy M Y 3 A B P 4 A M P B A E M L U P G G I N P L O E G R O P U P U S M A G R E E T E T N A U Q U Q 2 A D R L E N S 3 F C A 2 A 1 X 2 A Y A 2 1 S E L N S 2 A 2 L N E L A V V E B P M Sensors 3 PLC S B 3 D HSIS QUADRUPOLE PROFILE BEAM SWITCHING P B M s i n A u 2 e ) E STEERING LENS 1 LENS 2 Q1 LENS 3 STEERING MONITOR 1 PUMP MAGNET o n t ( 1 A A L V V BPM PUMP EXIT BEAM MAGNET X0 MAGNET X1 Y1 GROUP PUMP GROUP 2 BLV GROUP VALVE SWM 21 ˚ Z1 2Z Z-TGT TARGET NN 143G ISO Beam Room SBD 1 SBD 2 FARADAY 21 ˚ 48 ˚ SBD 3Z FARADAY CUP 1 CUP 2Z o n ( P B M B 2 B 1 U Q A D t n i u s e ) V L A E V B P M E L 1 N S Q 2 B R U L O P E Control BENDING MAGNET 7 L E N S 2 L E NN 143J D B S 3 S N 3 FIX Beam Room F C Control V 1 C A L V B 2 E V A B 2 L V G R O M P U E E B C P U P B S D B P M A P M G U L B S T E E 3 6 S E X P E M R I N E M R A G N E T G I N Q G P R O M U U P T A O I T A T N L Y G G X E N L S U A D R U Q P O L C C F 2 B 1 P 1 E L N S 2 3 B E E E L N S 3 C 2 V A L S X P E P U M V E T T A E M I R B S D 4 B G R P P U O O I N A T N 2 L B M P T A R T G E 8 B S 5 D UNIVERSITY OF WASHINGTON MEDICAL CENTER CNTS - SEATTLE CLINICAL CYCLOTRON DATE: 18JAN12 DRAWING NUMBER: SEA 8001 REVISED: DRAWN BY: SS TITLE: BEAMLINE LAYOUT Private Ethernet Programmable Logic Controller
CNTS Control Beamline Therapy Sensors PLC HSIS Control Control Prescription Safety: The beam will turn off and remain off if any machine setting goes out of prescribed tolerances.
CNTS Control Beamline Therapy Sensors PLC HSIS Control Control Over 30 year safety record! Prescription Safety: The beam will turn off and remain off if any machine setting goes out of prescribed tolerances.
Now: CNTS++ Beamline Therapy Sensors PLC HSIS Control Control Originally written in C. Want to extend treatment capabilities.
Now: CNTS++ New version in EPICS Beamline Therapy Sensors PLC HSIS Control Control++ EPICS documentation No formal definition The Maximize Severity attribute is one of NMS (Non-Maximize Severity), MS (Maximize Severity), MSS (Maximize Status and Severity) No type checking or MSI (Maximize Severity if Invalid). It determines whether alarm severity is propagated across links. If the attribute is MSI only a severity of INVALID_ALARM is propagated; settings of MS or MSS Highly dynamic propagate all alarms that are more severe than the record's current severity. For input links the alarm severity of the record referred to by the link is propagated to the record containing the link. For output Ubiquitous float pt. links the alarm severity of the record containing the link is propagated to the record referred to by the link. If the severity is changed the associated alarm status is set to LINK_ALARM, except if the attribute Config control flow is MSS when the alarm status will be copied along with the severity.
Now: CNTS++ Beamline Therapy Sensors PLC HSIS Control Control++ Prescription Safety? Will the beam will turn off and remain off if any machine setting goes out of prescribed tolerances?
Therac-25 radiation therapy machine Excessive radiation killed patients (1985-87) – New design removed hardware that prevents the electron- beam from operating in its high-energy mode. Now safety checks done in software. – Equipment control software task did not properly synchronize with the operator interface task, so race conditions occurred if the operator changed the setup too quickly. – Missed during testing because it took practice before operators worked quickly enough for the problem to occur.
Ariane 5 rocket (1996) Rocket self-destructed 37 seconds after launch – Cost: over $1 billion Reason: Undetected bug in control software – Conversion from 64-bit floating point to 16-bit signed integer caused an exception – The floating point number was larger than 32767 – Efficiency considerations led to the disabling of the exception handler, so program crashed, so rocket crashed
Mars Polar Lander Legs deployed à Sensor signal falsely indicated that the craft had touched down (130 feet above the surface) Then the descent engines shut down prematurely Error later traced to a single bad line of software code Why didn’t they blame the sensor?
More examples • Mariner I space probe (1962) • Microsoft Zune New Year’s Eve crash (2008) • iPhone alarm (2011) • Denver Airport baggage-handling system (1994) • Air-Traffic Control System in LA Airport (2004) • AT&T network outage (1990) • Northeast blackout (2003) • USS Yorktown Incapacitated (1997) • Intel Pentium floating point divide (1993) • Excel: 65,535 displays as 100,000 (2007) • Prius brakes and engine stalling (2005) • Soviet gas pipeline (1982) • Study linking national debt to slow growth (2010) • …
Software bugs cost money • 2013 Cambridge University study: Software bugs cost global economy $312 Billion per year – http://www.prweb.com/releases/2013/1/prweb10298185.htm • $440 million loss by Knight Capital Group in 30 minutes – August 2012 high-frequency trading error • $6 billion loss from 2003 blackout in NE USA & Canada – Software bug in alarm system in Ohio power control room
Building Quality Software What Affects Software Quality ? External Correctness Does it do what it supposed to do? Reliability Does it do it accurately all the time? Efficiency Does it do without excessive resources? Integrity Is it secure? Internal Portability Can I use it under different conditions? Maintainability Can I fix it? Flexibility Can I change it or extend it or reuse it? Quality Assurance (QA) – Process of uncovering problems and improving software quality – Testing is a major part of QA
Software Quality Assurance (QA) Testing plus other activities including: – Static analysis (assessing code without executing it) – Correctness proofs (theorems about program properties) – Code reviews (people reading each others’ code) – Software process (methodology for code development) – …and many other ways to find problems and increase confidence No single activity or approach can guarantee software quality “Beware of bugs in the above code; I have only proved it correct, not tried it.” -Donald Knuth, 1977
What can you learn from testing? “Program testing can be used to show the presence of bugs, but never to show their absence!” Edsgar Dijkstra Notes on Structured Programming, 1970 Nevertheless testing is essential. Why?
Recommend
More recommend
