CSE 331 Outline Software Design and Implementation • Why correct software matters – Motivates testing and more than testing, but now seems like a fine time for the discussion • Testing principles and strategies Lecture 8 – Purpose of testing – Kinds of testing Testing – Heuristics for good test suites – Black-box testing – Clear-box testing and coverage metrics – Regression testing Zach Tatlock / Spring 2018 Clinical Neutron Therapy System Non-outline • Modern development ecosystems have much built-in support for testing – Unit-testing frameworks like JUnit – Regression-testing frameworks connected to builds and version control – Continuous testing – … • No tool details covered here – See homework, section, internships, …
CNTS Beamline CNTS Control Hardwired Safety Interlock System NN 141C A C 3 F Vault G D 5 T I N S B O T A T P R E N Beam Monitor U M E M P U P E L L E R O U P O G M D R 3 A 2 B P U A A E N S Q Q 3 L 5 2 N S D 4 N G L E S B E R I 1 S T E E T N S G N L E M A X 3 A Focusing 3 A Y M B P NN 141C FC3A 4 Vault PUMP SBD 5 ROTATING ELEMENT Beamline QUADRUPOLE Q3A LENS 3 GROUP A2 BPM 5 LENS 1 LENS 2 SBD 4 STEERING X3A MAGNET Therapy 4 BPM Y3A A BEAM PLUG A STEERING U G Q2A QUADRUPOLE LENS 3 FC2A PUMP GROUP A1 MAGNET X2A Y2A P L LENS 1 LENS 2 VALVE A2 A M G BPM 3 M P B E R I N Sensors PLC HSIS Magnets P U P E E QUADRUPOLE Q1 PROFILE BEAM SWITCHING BPM 2A (not in use) A1 VALVE SBD 3 O U S T N E T BEAM EXIT STEERING X0 MAGNET LENS 1 LENS 2 LENS 3 MAGNET STEERING X1 Y1 MONITOR 1 SWM PUMP GROUP MAGNET 21 ˚ GROUP PUMP BPM 2 BLV Z-TGT GROUP PUMP O L E G R A G VALVE Z1 2Z TARGET ISO Beam Room NN 143G U P 1 M A SBD 1 SBD 2 CUP 1 FARADAY 21 ˚ 48 ˚ SBD 3Z FARADAY CUP 2Z D R 3 A X 2 BPM 2B (not in use) VALVE B1 BPM LENS 1 QUADRUPOLE Q2B U A A E N S C 2 A A Control MAGNET BENDING SBD 3 7 LENS 2 LENS 3 FIX Beam Room NN 143J Q Q 2 L F Y 2 C1 VALVE FC2B Control 2 GROUP PUMP B2 VALVE E N S C SBD 3 BPM 6 EXPERIMENTAL BEAM PLUG B MAGNET STEERING L 2 FC2C B1 GROUP PUMP STATION XG YG LENS 1 LENS 2 QUADRUPOLE Q3B 1 A VALVE C2 EXPERIMENTAL LENS 3 N S V E STATION SBD 4 GROUP B2 PUMP L E V A L TARGET 8 BPM SBD 5 M UNIVERSITY OF WASHINGTON MEDICAL CENTER B P DATE: 18JAN12 CNTS - SEATTLE DRAWING NUMBER: SEA 8001 CLINICAL CYCLOTRON REVISED: DRAWN BY: SS 3 TITLE: BEAMLINE LAYOUT D 3 S B BEAM 2 A QUADRUPOLE P M ) PROFILE SWITCHING B u s e t i n E Q1 ( n o L V MONITOR MAGNET V A BPM STEERING LENS 1 LENS 2 LENS 3 STEERING A 1 1 PUMP PUMP PUMP BEAM MAGNET MAGNET 2 GROUP GROUP EXIT X0 X1 Y1 GROUP BLV 21 ˚ VALVE SWM Z1 2Z Z-TGT TARGET NN 143G Private ISO Beam Room SBD 1 SBD 2 21 ˚ SBD 3Z FARADAY 48 ˚ FARADAY CUP 1 CUP 2Z B P M Q ( n o 2 B B 1 U A Ethernet t n i V A L D R U u s e L V B P E N P O ) E M S 1 Q 2 L E B BENDING 7 L E N Beam Plug NN 143J S 2 MAGNET L E N S 3 FIX Beam Room Cyclotron Faraday Cup S B D 3 F C V 2 B A C L 1 V E B 2 V A L V G P E R U M B C O S E A U P B B M P P D P L U M G B S T E 3 E X E R 6 P E M A I N G R I M G N S E N E T Q P U T A T T A X G U A M P O I N L D R G R O Y G L E N U P U P S 1 Q 3 O L E F B B C 1 L E 2 N S C 2 L E N S 3 E V X A C P L 2 E P U M V S R S B G P E A T I D 4 R O T M U P E B I O N 2 N T Vacuum Pump A L T A R B P G E T M 8 S B D 5 UNIVERSITY OF WASHINGTON MEDICAL CENTER Programmable Logic Controller CNTS - SEATTLE CLINICAL CYCLOTRON DATE: 18JAN12 DRAWING NUMBER: SEA 8001 REVISED: DRAWN BY: SS BEAMLINE LAYOUT TITLE: CNTS Control CNTS Control Beamline Beamline Therapy Therapy Sensors PLC Sensors PLC HSIS HSIS Control Control Control Control Over 30 year safety record! Prescription Safety: Prescription Safety: The beam will turn off and remain off The beam will turn off and if any machine setting goes out of remain off if any machine setting prescribed tolerances. goes out of prescribed tolerances.
Now: CNTS++ Now: CNTS++ New version in EPICS Beamline Beamline Therapy Therapy Sensors PLC HSIS Sensors PLC HSIS Control Control Control Control++ EPICS documentation No formal definition The Maximize Severity attribute is one of NMS (Non-Maximize Originally written in C. Severity), MS (Maximize Severity), MSS (Maximize Status and Severity) No type checking Want to extend treatment capabilities. or MSI (Maximize Severity if Invalid). It determines whether alarm severity is propagated across links. If the attribute is MSI only a severity of INVALID_ALARM is propagated; settings of MS or MSS Highly dynamic propagate all alarms that are more severe than the record's current severity. For input links the alarm severity of the record referred to by the link is propagated to the record containing the link. For output Ubiquitous float pt. links the alarm severity of the record containing the link is propagated to the record referred to by the link. If the severity is changed the associated alarm status is set to LINK_ALARM, except if the attribute Config control flow is MSS when the alarm status will be copied along with the severity. Now: CNTS++ Therac-25 radiation therapy machine Excessive radiation killed patients (1985-87) Beamline Therapy – New design removed hardware that prevents the electron- Sensors PLC HSIS Control Control++ beam from operating in its high-energy mode. Now safety checks done in software. – Equipment control software task did not properly synchronize with the operator interface task, so race Prescription Safety? conditions occurred if the operator changed the setup too quickly. Will the beam will turn off and remain – Missed during testing because it took practice before off if any machine setting goes out of operators worked quickly enough prescribed tolerances? for the problem to occur.
Ariane 5 rocket (1996) Mars Polar Lander Rocket self-destructed 37 seconds after launch Legs deployed à Sensor signal falsely indicated that the craft had – Cost: over $1 billion touched down (130 feet above the surface) Then the descent engines shut down prematurely Reason: Undetected bug in control software – Conversion from 64-bit floating point to 16-bit signed integer Error later traced to a single bad line of software code caused an exception Why didn’t they blame the sensor? – The floating point number was larger than 32767 – Efficiency considerations led to the disabling of the exception handler, so program crashed, so rocket crashed More examples Software bugs cost money • Mariner I space probe (1962) • 2013 Cambridge University study: Software bugs cost global economy $312 Billion per year • Microsoft Zune New Year’s Eve crash (2008) • iPhone alarm (2011) – http://www.prweb.com/releases/2013/1/prweb10298185.htm • Denver Airport baggage-handling system (1994) • Air-Traffic Control System in LA Airport (2004) • $440 million loss by Knight Capital Group in 30 minutes • AT&T network outage (1990) – August 2012 high-frequency trading error • Northeast blackout (2003) • USS Yorktown Incapacitated (1997) • $6 billion loss from 2003 blackout in NE USA & Canada • Intel Pentium floating point divide (1993) – Software bug in alarm system in Ohio power control room • Excel: 65,535 displays as 100,000 (2007) • Prius brakes and engine stalling (2005) • Soviet gas pipeline (1982) • Study linking national debt to slow growth (2010) • …
Building Quality Software Software Quality Assurance (QA) What Affects Software Quality ? Testing plus other activities including: External – Static analysis (assessing code without executing it) Correctness Does it do what it supposed to do? – Correctness proofs (theorems about program properties) Reliability Does it do it accurately all the time? – Code reviews (people reading each others’ code) Efficiency Does it do without excessive resources? – Software process (methodology for code development) Integrity Is it secure? – …and many other ways to find problems and increase confidence Internal Portability Can I use it under different conditions? Maintainability Can I fix it? No single activity or approach can guarantee software quality Flexibility Can I change it or extend it or reuse it? “Beware of bugs in the above code; I have only proved it correct, not tried it.” Quality Assurance (QA) -Donald Knuth, 1977 – Process of uncovering problems and improving software quality – Testing is a major part of QA What can you learn from testing? What Is Testing For? Validation = reasoning + testing “Program testing can be used to show – Make sure module does what it is specified to do the presence of bugs, but never to – Uncover problems, increase confidence show their absence!” Two rules: Edsgar Dijkstra 1. Do it early and often Notes on Structured Programming, 1970 – Catch bugs quickly, before they have a chance to hide – Automate the process wherever feasible 2. Be systematic – If you thrash about randomly, the bugs will hide in the corner until you're gone – Understand what has been tested for and what has not Nevertheless testing is essential. Why? – Have a strategy!
Recommend
More recommend
