CMMI Level 5: Return on Investment for Raytheon N TX Donna Freed Network Centric Systems, McKinney, TX
Achieving CMMI Level 5 • We did it! • How Did We Do It? – Achieve Engineering Goals. This presentation describes the benefits This presentation describes the benefits of achieving CMM Level 4 in 2001, and of achieving CMM Level 4 in 2001, and then to achieve CMMI Level 5 in 2003 . then to achieve CMMI Level 5 in 2003 . 12/1/2004 Page 2
We Did IT! – Raytheon North Texas is the first site in Raytheon and fifth company in the world to achieve CMMI Level 5. – Measurable results are achieved before achieving Level 5. – This Presentation shows the actual ROI of going to each level, as well as our ROI projection. 12/1/2004 Page 3
How Did We Use CMMI to Achieve? • Why is Raytheon North Texas pro-active about achieving CMMI Level 5? – Because we want to achieve the performance excellence goals required by our business. We are focused on achieving performance excellence and recognition as the preferred supplier for new business. 12/1/2004 Page 4
Envision Improvement Integrated Product Integrated Product Teams: Development System: Cross-functional resources to IPTs Where we define our product implement our processes development processes Raytheon Six Sigma: Capability Maturity How we improve our Model Integration: processes The yardstick for Programs Integrate R6 σ , IPDS judging the maturity of and CMMI into their Pland our processes IPTs IPTs Product teams use common tools and processes in an environment of continuous improvement guided by industry “Best Practices” 12/1/2004 Page 5
How Did We Achieve Performance Goals? How did we use CMMI to achieve our performance excellence goals? – We picked performance goals that were important to us. – The metrics data we collect characterizes the organizational performance in terms of our organizational goals and identifies opportunities of improvement. 12/1/2004 Page 6
SWEC SWIP Objectives • Meet Commitments (to Customer) – Intent: Meet the cost and schedule objectives of the programs we support. – Quantification: CPI and SPI • SW Price – Intent: Price software engineering products competitively – Quantification: $ / DLOC • Deliver Quality – Intent: Deliver quality software engineering products – Quantification: In-phase Defects and Defect Density We have been executing statistical process control on the overall process using these measures for years. 12/1/2004 Page 7
Organization Process Analysis Process Capability for CPI Process Tooling People M ultiple data Too program Subjectivity Procedure sources Tools & (differing points focused, need orig phases of for guidelines Clear product line & process to of view) LSL = 0.975, Nominal = 1.0, USL = 1.15 Doors Continuus Defect Logger Doublecounts?? Using correct autom ate org view people in reviews defect Source Types Reason code Defect Existence Phase of origin 24 capture Experts Limited Domain SW Experts Systems Cp = 0.09 W hat counts How to count Overloaded (definition of defects) defects in Attrition reused code in counting differences Pjt to pjt 20 Tools not user No com m on Cpk = -0.03 friendly repository for Type & reason Tim e to do People shared code codes not Defect curve Continuus learning frequency prework concerned 16 Peer review process useful about defect Cpk (upper) = 0.2 Constraints Schedule Round trip eng. counts being variation Criteria types Varying Inconsistent Not a priority (i.e. integrated used against Defect on a program tooling) them Containm ent 12 Cpk (lower) = -0. M etric has Excessive People need to Peer review training Custom er care Variation: s > 28% know the concept abouts Cr = 11.61 Mix of Purpose not clear of operations 8 No Org Training JIT Severity Needs to include: program s Always start at New people do not Type & Domain Variation Old/New 100% know how to reason Cpm = 0.09 classify defects 4 Phases Effective m eeting training Roles K = -0.61 needed for peer reviews 0 Conducting People don't know how to Meeting & use the defect data 0 0.5 1 1.5 2 2.5 3 follow-up Corrective action CPI Training Program s $/DLOC Regression Model SW Program Duration (M onths) 400 Dollars per DLOC > 48 Mean = 23 >42 and <=48 300 Median = 18 >36 and <=42 Min = 5 Max =120 >30 and <=36 STD EV = 22 200 n = 55 >24 and <=30 >18 and <=24 100 > 12 and <=18 > 6 and <= 12 0 <= 6 0 0.2 0.4 0.6 0.8 1 0 5 10 15 20 ELOC/DLOC Use R6 σ Tools for Metrics Analysis
Improvement Results • Demonstrated the linkage between R6 σ and CMMI Levels 4 & 5. • Characterization included over 300 applications of R6 σ tools such as ANOVA, cause and effect, regression analysis, histograms, Cpk, hypothesis testing, logical process mapping, and others. • Identified five projects to reduce variation in organizational performance and support the CMMI Level 5 timeline. • Enabled CMMI Level 5 certification. – Improvement of Business Performance was recognized by Assessment Team as global strength in the CMMI Level 5 Assessment . • Contributed ROI of 3:1 through significant cost avoidance realized by organization improvements 12/1/2004 Page 9
Operational Results • Achieving CMMI Level 5 Certification for Raytheon image and competitive advantage is one thing, but look at the operational results. • “Meeting Commitments” all improved concurrent with SEI CMMI Level 5 certification Across the organization, we improved: – CPI by 5 percentage points, and reduced variation by 34%. – SPI by 8 percentage points, and reduced variation by 50% – Defect Density by 44 percentage points, and reduced variation by 31% 12/1/2004 Page 10
Cost Performance Index Process Capability for CPI 40 frequency January 2004 30 20 10 0 CPI Process Capability for CPI 10 February 2001 frequency 8 6 4 2 0 CPI We no longer have CPI special cause variation on the low end! 12/1/2004 Page 11
Cost Performance Index SWEC CPI Trend CMM Level 4 CMMI Level 5 June 2001 September 2003 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Q Q Q Q Q Q Q Q y l b r r p c u a p a e 1 2 3 4 1 2 3 4 e e J M A M D F S Upper DROV Nominal value Lower DROV Mean Performance Improved CPI by 5 percentage points, and reduced variation by 34%. 12/1/2004 Page 12
Schedule Performance Index Process Capability for SPI 30 January 2004 frequency 25 20 15 10 5 0 SPI Process Capability for SPI 15 February 2001 frequency 12 9 6 3 0 SPI We are still moving in the right direction! 12/1/2004 Page 13
Schedule Performance Index SWEC SPI Trend CMM Level 4 CMMI Level 5 June 2001 September 2003 May 01 Jul 01 Feb 01 Mar 01 Sep 01 Dec 01 1Q02 2Q02 3Q02 4Q02 1Q03 2Q03 3Q03 4Q03 Apr 01 Upper DROV Nominal value Lower DROV Mean Performance Improved SPI by 8 percentage points, and reduced variation by 50% 12/1/2004 Page 14
Defect Density Process Capability for Defect Density 60 January 2004 frequency 50 40 30 20 10 0 Defect Density Process Capability for Defect Density 4 February 2001 frequency 3 2 1 0 Defect Density Continuing improvement in mean and variation. 12/1/2004 Page 15
Defect Density SWEC Defect Density Trend CMM Level 4 CMMI Level 5 June 2001 September 2003 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Q Q Q Q Q Q Q Q y l r b r p c u a p a e e e 1 2 3 4 1 2 3 4 J M A M D F S Upper DROV Nominal value Lower DROV Mean Performance Improved Defect Density by 44 percentage points, and reduced variation by 31% 12/1/2004 Page 16
Characterize – CPI Analysis Plot of Regression Model CPI Percentage Organization Process Adherence � Regression Analysis included a sample of various process characteristics. � Projects that follow the standard process tend to have a better and more predictable CPI performance. � Process adherence is not a guarantee of CPI success. It improves the probability of CPI success. 12/1/2004 Page 17
Characterize – SPI Analysis Predicted SPI based on Process Adherence vs. Observed SPI Observed SPI 0 0 Multiple Regression Analsysis Prediction of SPI � Multiple Regression Analysis included Level 2, Level 3, and Level 4 process characteristics. � Projects that follow the standard process tend to have a better and more predictable SPI performance. � Process adherence is not a guarantee of SPI success. It improves the probability of SPI success. � Organizational process adherence is the only identified factor affecting SPI. 12/1/2004 Page 18
Results • Our improvements were recognized as organizational strengths in the appraisal. “This accomplishment leads the way for Raytheon to distinguish ourselves from the competition and achieve customer satisfaction through superior program execution. There is no higher illustration of customer focus than this level of excellence.” Colin Schottlaender, Raytheon NCS President These improvements contributed to ROI of 3:1 12/1/2004 Page 19
Recommend
More recommend
