Cost Overruns and Their Precursors : An Empirical Examination of - PowerPoint PPT Presentation

Cost Overruns and Their Precursors : An Empirical Examination of Major DoD Acquisition Programs International Cost Estimating and Analysis Association Denver, CO 10-13 June 2014 Alan K. Gideon, P.E. Dr. James Wasek Dr. Enrique Campos-Nañez Dr. Pavel Fomin The School of Engineering and Applied Science of The George Washington University in partial satisfaction of the requirements for the degree of Doctor of Philosophy

The Persistence of the Problem • Trends across the 95% CI for the Mean wider commodity list 3.0 Combat Aircraft/Missiles improved into the Aircraft/Ships 1990’s ( Younssi) 2.5 • Aircraft remained 2.0 relatively immune to improvement 1.5 • Graphic does not include outliers 1.0 0.5 `A million dollars here, and a million dollars * From Younossi, et al, there, and pretty soon, gentlemen, you`re talking using a wider group of commodities about real money.` Attributed to Senator E. Dirksen Cost overruns remain a serious problem 2

Previous Approaches to the Problem • Cost and schedule overruns are not a new problem • Previous work Has tended to cast “cost overrun” as an amorphous lump, or o Investigators have dug deeper into the details of their specialties o • Previous papers and policy changes have failed to resolve the issue o RAND Inadequate initial funding Unexpected technical difficulties Requirement changes Estimating errors Cost growth ~ ƒ (quantity purchased) (Dews et al. 1979) o IDA added Supply, labor shortages Concurrency Force majeur Cost growth ~ ƒ (median domain growth rates) (Asher and Maggelet 1984) o WSARA 2009, updates to DoDI 5000 series, lower level directives (P.L. 111-23) Previous approaches have addressed symptoms of the basic question 3

Technical Risk as a Precursor to Cost • There are no truly independent variables: • Programmatic/Business Contract Changes • Technical Technical/Performance • Schedule Schedule • Cost Cost “All roads lead to Rome”, and additional cost 4

Decisions, Decisions, Decisions… Systems Engineering Technical Reviews A B C Production & Technology Development Operations & Engineering & Manufacturing Development Material Solution Deployment Support System Systems Analysis (MSA) LRIP/ IOT&E /Full Rate Requirements & Architecture & Sustainment & System Design System Demonstration Material Production & Deployment Technology Technology Disposal Post Post Development Development Demonstration Decision PDR A CDR A FRP Decision Review 1 2 3 4 5 6 Annual Sufficiency Reviews Pass 1 Pass 2 TRA SSR IRR TRA ITR ASR SRR I SRR II SFR PDR 2 IBR PDR 12 CDR 2 PCA ISR TRR FRR OTRR SVR/ FCA/PRR Preferred System System Allocated Product Product 1 PDR Closure System Specification Functional Baseline Baseline Baseline 2 For ships, PDR and CDR to take Concept Baseline place prior to MS B Notice the caveat From: Naval Sea Systems Engineering Technical Review Handbook Work scope and costs are tied to Milestone decisions 5

The Cost Prediction Initialization Point • It is important to note a significant normally unstated difference between the acquisition of ships and the acquisition of other customized purchases the Department of Defense makes • We don’t build prototype ships Outcomes occasionally notwithstanding, the intent is that every ship built for o the U.S. Navy will become an operational asset. This affects the definition of “baseline cost”, used later o Significant work scope and costs begin before MS B for ships 6

Knowing the Neighborhood • Metaphorically speaking, the more interesting destinations sometimes pass through or near some bad neighborhoods – creating risks o Cox paper Joint Confidence Level Scatterplot o Does not show confidence levels o “Grade inflation” o Cannot show $800K performance to plan Total Cost $700K Risk “Cube” (Matrix) $600K 1 2 3 4 5 $500K $400K Probability $300K $200K $100K 50 65 80 95 110 Months Duration 1 2 3 4 5 (From: Rippe, Hogan, Elliot 2011) 7 Consequence

How Bad Can it Get? 5.0K 4.5K • Like asking how low a 4.0K NASDAQ Composite (^IXIC) 3.5K particular stock price can go 3.0K 2.5K 2.0K 1.5K 1.0K 0.5K 1990 1992 1994 1996 1998 2000 2002 • Sound decisions can only be made with sound information 0.4 6000 0.3 5000 Probability $M (BY14) 4000 0.2 3000 0.1 2000 1000 0.0 1 2 3 4 0 Program Cost Ratio Program Cost Ratio 1 2 3 4 5 6 7 8 9 Years Since MS 0 Sound program and portfolio decisions require solid data, sound analysis 8

The Cost Risk Box Canyon • Markowitz “portfolio effect” Risk is minimized through o diversification Requires that assets be truly o independent Presumes investors are rational o • DoD 7000.14R: recommends budgeting to the most probable cost • DAPA Report 2006: recommended an 80% confidence level • DTM 09-027 (5)(e): requires justification if the recommended confidence level is less than 80% • Possible maximum values associated with violating these “most probable costs” is not part of anyone’s spreadsheet. Official policy is at odds with program behavior and decision patterns 9

Avoiding the Box Canyon • Smart Reminded us of the “flaw of averages” o Value at Risk: “the maximum loss not exceeded with a given probability” o Recommended lognormal v. normal distribution for lower risk o Conditional Tail Expectation o • “Conspiracy of hope” percentile funding is, unfortunately, built on faulty logic and does not work • The way an aviator avoids becoming another “box canyon statistic” is by not flying into them “Six months after winning a coveted $35 billion aerial tanker contract, Boeing Co. announced last year that the first planes would cost $1 billion more than promised during the contract’s competition. “ CQ WEEKLY – IN FOCUS, Jan. 21, 2012 Avoiding box canyons requires adopting different decision inputs 10

Five Year Family Tendencies Unlike previous approaches • We limit ourselves to a five year “crystal ball” Not claiming to see too far o into the future Consistent with the needs of o the Five Year Defense Plan • Add two more factors Difficulty of the task to be o performed Funding dedicated to risk o mitigation • Different points of reference • Obviously different outcome spectra where y = years between program approval and IOC 11 0 = Program approval point

Distribution of Five Year Cost Ratios Number of Instances 1.0 1.5 2.0 2.5 3.0 Ratio of 5yr Cost to Initial Cost Different Outcomes Imply Different Input Details 12

The Leading Edge of Technology • Estimates for “modest” improvements are more accurate • No penalty for under- estimating costs • ~1970 marks the availability of greater computing power Engine design o Reduced RCS o • Aircraft were divided into three groups Pre-1970 o Post 1970 o Derivatives & special o cases All data taken from open sources Computing power has made significant improvements possible 13

The Leading Edge of Technology • ~1970 marks the availability of greater computing power Engine design o Reduced RCS o • Ships were divided into three groups Pre-1970 o Post 1970 o Intentionally avoided “cutting Derivatives o edge” performance in favor of greater reliability All data taken from open sources Some progress was being made before significant computing improvement 14

Results to Date: Aircraft Combat Aircraft 5 yr Cost per 3-Variable Model 30000 R 2 = 99.12% R 2 (adj) = 98.68% 25000 S = 967.3 Data Fitted per Model 20000 15000 10000 Analysis of Variance Variable P 5000 Cost @ MS B 0.0000 Tech risk 0.7304 (RDTE/Cost) B 0.3396 0 0 5000 10000 15000 20000 25000 30000 Actual 5 yr Q-A Cost Cost| 5yr = ƒ (domain tendencies, tech risk, [RDTE/Q-A Cost] 0 ) 15

Results to Date: Ships Combatant Ship 5 yr Cost per 3-Variable Model 40000 R 2 = 93.36% R 2 (adj) = 83.40% S = .8023 • Not quite as good, but respectable 30000 Data Fitted per Model • Johnson transform required 20000 Analysis of Variance 10000 Variable P Cost @ MS 0 0.6801 Tech risk 0.9389 0 (RDTE/Cost) B 0.0951 0 10000 20000 30000 40000 Actual 5 yr Q-A Cost Cost| 5yr = ƒ (domain tendencies, tech risk, [RDTE/Q-A Cost] 0 ) 16

Using the Asher-Maggelet Approach: Aircraft where y = years between program approval and IOC 0 = Program approval point P = 0.0000000

Using the Asher-Maggelet Approach: Ships where y = years between program approval and IOC 0 = Program approval point P = 0.0656