Testing Lessons Learned and Suggested Improvements for Rapid - PowerPoint PPT Presentation

U.S. ARMY WHITE SANDS MISSILE RANGE Testing Lessons Learned and Suggested Improvements for Rapid Prototyping and Acquisition Mr. Jerry Tyree, Deputy Commander / Tech Director White Sands Test Center 18 January 2017

Outline 1. Testing in Rapid Acquisition – Criteria Based 2. Risk Management Approach 3. Test Design, Planning, Execution 4. Test Execution and Analysis 5. Training (DOTLmPF) in Testing 6. Threat Environment / Presentation 7. Data Acquisition / Instrumentation 8. Test Results, Analysis, Reporting 9. Prototype Testing in Rapid Acquisition 10. Summary 2

Testing As a Solution to Acquisition Com om m on on Acqu quisit ion on Obj bj ect ive Provide Warfighter Capability to Defeat the Threat and Defend the Nation - Quality Performance, Responsive, Cost Effective Testing Is a LEARNING Tool to Inform & Enables Warfighter Effectiveness Survivability, Suitability   Research & development of technologies Develop & validate Warfighter capabilities   Exploit threat systems, capabilities, operations Develop & validate tactics, procedures, training Acquisition Life Cycle Effe ffect ct ive T Test st ing Notional Resources Over Time • Finds / Solves Issues Life Cycle Cost is highly • Characterizes Performance dependent on the T&E Phase CUMULATIVE COST & RESOURCES CUMULATIVE COST & RESOURCES • Is Relevant to the Threat Environment Finding & Fixing issue Upfront • Includes all aspects of DOTLmPF Saves Lives, Cost Later - Reliability • Informs Risk - Repairs or rework - Maintenance, spares Test occurs throughout life cycle • Numerous Studies over the last 2 decades (stockpile reliability, product improvements, threat…) PROCURE PROCURE • Aimed at Cost, Schedule, OT failure DESIGN / DESIGN / Prototype Prototype & FIELD & FIELD Operate / Maintain / Sustain – Improve – Operate / Maintain / Sustain – Improve – • Early Warfighter Involvement Adjust to threat & technology changes Adjust to threat & technology changes • Reliability, Readiness, Schedule COEA COEA AoA / AoA / TEST TEST TIME TIME • Combine DT/OT 3

Testing Rapid Acquisition “ Historical Faster Testing Approach” Criteria Based Acquisition Drives • Criteria Based – Schedule Driven Funding Constrained Criteria Based Test and Evaluation • Generally Sequential Process by Design – Not Combat M&S Material Solution to Integrated > COEA / BOI Operational Rqmt SE M&S • Test Type “Tools” – Generally Phase / Criteria Driven Design to Spec > Spec Criteria • Generally Binary – PASS / FAIL Fab. Prototype Ctr Confidence DT Spec / Function CURRENT PROCESS IS NOT TIED TO MISSION RISK DT&E Criteria Ver/Val TTP / Log Train FDT&E Log Capabilities & Limitations Ver/Val Demo FOA Faster by Less of Each Operational OT&E Criteria Ver/Val • Integration / Interoperability / Compatibility Procure Report • Leader / Operator Training Field Criteria Train Met? Sustain 4

An Integrated Risk Assessment Approach Integrated Risk Management Concept Fundamentals – Test Design Key Challenges Les esson L Lea earned ed • How Much of Which Tool(s) will Provide • Criteria based requirements drive a pass/fail process – the Required Information? • Reliability drives schedule - M&S, DT, FDTE, OTE • Threat changes faster than our process can respond to (Cyber • How Much is Enough Information? example in software blocking) Safety & OT are Only Test Required by Law • Capabilities evolve over time – rarely work right / meet Sample Size, Reliability requirements the first time Rec ecom m en endat ion • Informed Risk Management Approach • Focus process Test Design and resources on Risk based requirements 1. Identified Risk • Lives/Impact, Performance, Cost • Risk to schedule (not fielding a capability against a threat) 2. Knowledge Required • What we know • What we don’t know • What is worth knowing (CBA) • What can be learned over time • Apply Knowledge Management and track or assess risk throughout 5

Rapid Acquisition T&E Test Design, Planning, Execution Lessons Learned: • Too much time repeatedly spent coordinating, socializing and mustering resources for test • Environmental, Safety procedures • Instrumentation Data Collection and Reduction • Threat Systems / Environment • Systems or Systems of Systems (acquisition, transportation, configuration) • Test is a learning continuum, capabilities evolve over time – changes in threat, technology, tactics, cyber Readiness Recommendation: Time • Establish persistent SME stakeholder teams – Consolidate and Co-locate • Knowledge grows/evolves over time improves quality, throughput, responsiveness • ALL stakeholders must be represented (materiel developer, combat developer (warfighter), tester, threat) • Establish persistent Test Bed - fall in • Representative systems of systems (networks) • Consistent instrumentation • Threat representation (threat systems, denied environments, • Climatic, dynamics, electromagnetic, logistics… 6

Rapid Acquisition T&E Test Execution and Analysis Lessons Learned: • Insufficient Instrumentation and Data Collection leads to re-test or indeterminate solutions • Constrains or cripples forensics required for efficient Test-Fix-Test • Post test data analysis leads to re-test, delays or inclusive evaluation • Sharing, communications and knowledge of all stakeholders is critical to rapidly fixing or assessing capabilities Recommendation: • Upfront, establish rapid, reconfigurable instrumentation and data collection • System /Threat Level - data links, emissions, optics, effects, signatures • Operations Level – Optics, Radiometric, TM, RF spectrum, Radar, Meteorological, GPS, • Require Real-Time Data Analysis and Display • Critical to test execution, refinement and execution decision and learning process • Must be re-configurable, acceptance of multiple inputs and custom presentation • Establish a “Joint” Analysis Team comprised of a lead by phase but include all stakeholders (PM, OEM, Combat Developer, Logistics, Training, Test/Evaluator) 7

Rapid Acquisition T&E Training in Value in Test, Test Value in Training Lessons Learned: • Early Prototypes and Rapid Acquisition Systems often have ‘engineering’ level or poor user interfaces • Training with the Lack of or Insufficient Experience will result in poor or failed test • Major difference in Training and Experience – can skew measures and results Recommendation: • Include Human Factors and Training Upfront and Continually • Warfighter involvement in test process results in matured training products and knowledgeable operators • Can mitigate or simultaneously satisfy the need for separate training/logistics test and data collection • Evolves with system maturity and development 8

DT&E “Success” > OT&E “Failure” Contributing Factor - Prototype vs. Production Rep SUT COMMON FACTS & EFFECTS M&S Combat Milestone B – COEA (there are exceptions) Effectiveness rqmts validation / spec • Schedule & Funding drive Acq Community to regard M&S Design – Design / mature design hand built EMD Prototypes as Production Rep component / concept System / SoS Result: Test design for full spectrum for world wide environments requires “sealed” systems Prototype (s) Fab /form/fit • Fixing “sealed” systems is more expensive, time Contractor consuming or impractical to support test/fix/test Confidence Test Result: Tendency to defer fixes or tests = Increased DT&E Prototypes Test – Fix Test risk to OT. Increased cost and schedule. Tendency to • Tech. Perf. Specs Cycle mod only certain prototypes resulting in various • World-wide environments configurations / fixes = risk to test success • Threat / Vulnerability • Tendency for “special treatment” and allowances for • Reliability Indicators limited prototype resources “Pass DT” Result: Fewer “operational” type reliability and durability issues are discovered LUT Prototypes LRIP • Prototypes get high use / use rates in excess of • limited number / spares IOT&E • various configurations OMS/MP estimates (through multiple environments) • delayed fixes to issues in DT Result: Increased risk of failure in LUT / OT • excessively used devices 9