Phase Containment Background: Mid-1990s Motorola Transmission - PDF document

Phase Containment Background: Mid-1990s Motorola Transmission Products Business Unit Effectivity Revenues $220 million / year “Cash cow” for the division About 100 software engineers; multi-sited Boston Spin Feb. 17, 2004 Two major telecommunications product lines “Box” products for enterprise interconnect Embedded software, multi - 68xxx processors Custom digital signal processor technologies Ross Seider >90% of value in software On-Fire Associates Release projects lasted 9 – 12 months On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution How phase containment initiative Business Unit Development Maturity was “sold” Previously achieved CMM Level 2 assessment Expected benefits of initiative Code reviews were a standard practice More diligence on design work products Avoiding mistakes (vs. containment) QA was embedded within design team, but reported Smoother system test phase to another organization Unexpected benefits of initiative Testing was “owned” by engineering Early visibility into project’s outcome Better precision on release criteria Mgmt. searching for ways to continuously improve (test coverage, outstanding defects, code stability) On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution Cost to fix problems Standard of Performance (Based on when the defect is created versus when its discovered) Requirements Specifications Complete Costs Unambiguous Work Review points Products Testable Traceable Code review Requirements specification Concise Design review Architecture spec Architecture review Independent of implementation Design spec Requirements review Conformant to internal and external standards Code On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution

Phase Containment Metric and Process Reviews Metrics per review Project Reliant Release 4.0 "November 21, 2001 Number of people Elapsed time Phase Review Points Work product size Defect volume Requirements Architecture Design Code Phase Defect severity, location Defect root cause Work Products Review Review Review Rev iew Module Integration Alpha Total Total Total Containment Test Test Test Errors Defects Faults Effectiv ity Requirements Specification 6 9 8 1 1 2 1 6 22 28 0.21 Interpretation of results Architecture Spec 130 30 9 14 12 16 130 81 211 0.62 Design Spec 156 40 25 23 12 156 100 256 0.61 Code 117 43 13 6 117 62 179 0.65 Review thoroughness and quality Test Spec 24 5 2 24 7 31 0.77 Faults by Phase 6 139 194 167 107 55 37 Project state Measured Historical In-process Defects 272 Project compared to similar projects In process Faults 705 Project management responses On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution Another value of CMM activity General Dynamics Decisions Systems Smoother entry into final testing (360 software engineers) Percent Phase CRUD Productivity Concise requirements (testability) CMM Level rework containment density per X factor effectivity KSLOC (relative) Review-able final test planning 2 23.2% 25.5% 3.2 1 x Agreement on test sequencing vs. build 3 14.3% 41.5% .9 2x Agreement on regression “sampling” Agreement on coverage objectives 4 9.5% 62.3% .22 1.9 x 5 6.8% 87.3% .19 2.9 x Sour ce: King, Diaz, Crosstalk: The Journal of Defense Software Engineering March 2002 On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution System Test Coverage Template Better final test planning Build 1 Build 2 Build 3 Build 4 Build 5 Build 6 Build 7 Build 8 Product Validation Test Start 1/15/03 2/10/03 4/10/03 4/15/03 4/25/03 5/10/03 5/15/03 6/1/03 Departm ent End 2/15/03 3/15/03 4/30/03 4/25/03 5/5/03 5/25/03 5/30/03 6/15/03 A ug-03 SDK Release 3.1 Cumulative coverage 0.10 0.14 0.32 0.54 0.63 0.67 0.77 0.78 Avail days 31 33 20 10 10 15 15 14 Total test cases Pass 6 3 9 21 11 5 10 2 Plan 125 Fail 1 2 12 6 2 1 1 0 Summary for all Tests revision 2.1 Total est. days B lock 2 3 5 7 5 0 1 1 196.5 To do 3 1 1 6 0 0 0 0 “ The plan is nothing, planning is everything” # Test cases this build 12 9 27 40 18 6 12 3 Planned P lanned P lanned P lanned P lanned P lanned P lanned P lanned Test plan section Test description Estim ated days coverage coverage coverage coverage coverage coverage coverage coverage Section 1 Totals Coverage in this build 20 0.44 0.28 0.39 0.17 0.11 0.11 0.06 0.06 Core Cum ulative coverage to date 0.44 0.67 0.78 0.78 0.78 0.89 0.94 1.00 Section 2 Totals Coverage in this build 40 0.20 0.20 0.10 0.35 0.30 0.15 0.10 0.10 Dwight David Eisenhower Comm Cum ulative coverage to date 0.20 0.30 0.40 0.55 0.65 0.80 0.90 0.95 Section 3 Totals Coverage in this build 58 0.00 0.00 0.60 0.53 0.07 0.00 0.00 0.00 Backend Cum ulative coverage to date 0.00 0.00 0.60 0.93 1.00 1.00 1.00 1.00 Section 4 Totals Coverage in this build 24 0.00 0.00 0.00 0.30 0.25 0.00 0.00 0.00 F rontend Cum ulative coverage to date 0.00 0.00 0.00 0.30 0.55 0.55 0.55 0.55 Section 5 Totals Coverage in this build 27 0.00 0.00 0.50 0.44 0.22 0.06 0.00 0.00 Backup Cum ulative coverage to date 0.00 0.00 0.50 0.78 1.00 1.00 1.00 1.00 Section 6 Totals Coverage in this build 27.5 0.00 0.00 0.00 0.42 0.00 0.00 0.47 0.00 On-Fire Associates Excellence in software execution Redundancy Cum ulative coverage to date On-Fire Associates Excellence in software execution 0.00 0.00 0.00 0.42 0.42 0.42 0.89 0.89

System Test Coverage Tool Test Coverage Decision Support (Build 4 “exploded”) Current status SDK 3.1 Test Coverage by Build Build 4 vs. Build 5 Coverage Analysis 25 0.9 1.2 0.8 Goal 20 1 0.7 Section 2: Core test 0.6 0.8 Coverage 15 suites 0.5 0.6 0.4 10 0.3 0.4 0.2 5 0.2 Section 3: Backend test 0.1 0 Backend Redundanc 0 0 Core Comm Frontend Backup y suites Build 1 Build 2 Build 3 Build 4 Build 5 Build 6 Build 7 Build 8 Build 9 Build 10 Build 11 Build 12 pass fail block Coverage On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution Did it matter ? “Innovation is creativity that ships” Steve Jobs “One solid measurement is worth a thousand expert opinions” Higher quality and more predictable releases Early visibility into project’s likely outcome (after the first few iterations) Consistency was important Postmortems found engineers were more effective and happier On-Fire Associates Excellence in software execution On-Fire Associates Excellence in software execution