Sensitivity Analysis using Experimental Design in Ballistic Missile - PowerPoint PPT Presentation

Sensitivity Analysis using Experimental Design in Ballistic Missile Defense Jacqueline K. Telford Johns Hopkins University Applied Physics Laboratory Laurel, Maryland jacqueline.telford@jhuapl.edu

Introduction Experimental design is used so that: • valid results from a study are obtained • with the maximum amount of information • at a minimum of experimental material and labor (in our case, number of runs). Poorly designed experiment Well designed experiment 150 runs (30 design points, 128 runs (all at different each repeated 5 times) design points) ⇒ 11 estimates of effects ⇒ 66 estimates of effects ⇒ 11/150 = 7% efficiency ⇒ 66/128 = 52% efficiency 2

Historical Perspective • The fundamental principles of experimental design are due primarily to R. A. Fisher, who developed them from 1919 to 1930 in the planning of agricultural field experiments at the Rothamsted Experimental Station in England. – Replication “To call in the statistician after the experiment is done – Randomization may be no more than – Blocking asking him to perform a postmortem examination; – Analysis Methods he may be able to say what – Factorial Designs the experiment died of.” 3

Sensitivity Analysis Overall Goal Provide a quantitative basis for assessing technology needs for missile defense architectures. 1. Screening Experiment Use experimental design to identify the main performance drivers in the scenarios from among the many possible drivers. 2. Response Surface Experiment Use experimental design to determine the shape (linear or curved) of the effects and interactions between the effects on the response variable for the main drivers to performance. 4

Process Scenarios The Answers... NEA SWA . 8 0 . 7 0 . 0 6 5 . 0 4 . 0 3 . 0 0 2 . 1 . 0 4 5 . 3 5 2 . 3 1 . 5 2 0 . 5 1 0 0 1 . 0 - Main and Two-Factor Simulation EADSIM Sensitivities Program MOEs: 1) FoS Protection Effect. 2 1.8 2) Inventory Usage(s) 1.6 1 1 1 - 1 1 1 - 1 1 - 1 - 1 - 1 1 - 1 - 1 1 - 1 1 - 1 1 1 - 1 - 1 1 1.4 Weapon System 1.2 2 1 - 1 1 - 1 - 1 1 1 1 - 1 - 1 - 1 1 - 1 - 1 1 1 - 1 1 - 1 - 1 1 - 1 1 3 1 1 - 1 1 - 1 - 1 - 1 1 - 1 1 - 1 - 1 - 1 1 - 1 - 1 - 1 1 - 1 1 - 1 1 Parameters to Screen* 0.8 4 1 - 1 - 1 - 1 1 1 - 1 1 1 1 - 1 - 1 - 1 - 1 1 1 - 1 - 1 - 1 1 1 1 0.6 0.4 5 - 1 - 1 - 1 1 1 1 1 - 1 1 - 1 1 - 1 1 - 1 - 1 1 - 1 1 - 1 1 - 1 1 0.2 6SDFH�%DVHG 6 1 1 1 - 1 - 1 - 1 1 - 1 1 - 1 1 - 1 - 1 1 1 1 - 1 1 1 - 1 1 1 0 0 0.2 0.4 0.6 0.8 1 Statistical 7 1 1 1 1 - 1 1 X Parameters 1 1 1 1 1 - 1 1 - 1 1 - 1 1 - 1 - 1 - 1 - 1 1 &XHLQJ 8 - 1 1 - 1 1 1 - 1 1 - 1 - 1 1 - 1 - 1 - 1 1 1 - 1 1 1 1 1 - 1 - 1 • Pfa Confidence 9 - 1 - 1 1 - 1 1 - 1 1 - 1 1 - 1 1 1 1 1 1 - 1 - 1 1 - 1 1 - 1 - 1 • Detection Sensitivity 1 0 - 1 - 1 1 - 1 1 - 1 1 - 1 1 - 1 - 1 1 - 1 - 1 1 1 1 1 1 1 1 - 1 1 1 1 1 1 1 1 - 1 1 - 1 - 1 - 1 - 1 - 1 1 1 - 1 1 - 1 1 1 - 1 - 1 1 • Time Track Sent 1 2 1 1 - 1 - 1 1 - 1 - 1 - 1 1 - 1 1 1 - 1 - 1 - 1 1 1 1 - 1 1 - 1 - 1 TMD FoS Architectural 1 3 - 1 1 1 1 - 1 - 1 - 1 1 - 1 1 - 1 - 1 1 1 - 1 - 1 - 1 1 - 1 1 1 - 1 Y Experiments • Track Accuracy 1 4 - 1 - 1 1 1 - 1 1 - 1 1 - 1 1 - 1 1 - 1 1 - 1 1 1 - 1 - 1 1 1 - 1 Performance Drivers 1 5 - 1 - 1 1 - 1 - 1 1 1 - 1 1 1 - 1 1 1 - 1 - 1 1 - 1 1 - 1 - 1 1 1 for the current NEA and 7KDDG • Launch Reliability SWA scenarios ,QWHUFHSWRU 1 9 5 1 1 - 1 - 1 - 1 1 1 1 1 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 1 - 1 1 - 1 • Reaction Time 1 9 6 - 1 1 - 1 - 1 - 1 1 1 - 1 - 1 - 1 1 1 1 1 1 1 1 1 - 1 - 1 1 - 1 1 9 7 1 1 1 1 - 1 1 1 - 1 1 - 1 1 - 1 1 - 1 1 1 1 1 - 1 - 1 - 1 - 1 • Boost Reliability 1 9 8 1 1 - 1 - 1 1 - 1 1 1 - 1 1 - 1 1 1 - 1 - 1 1 - 1 1 1 - 1 1 1 1 9 9 1 1 1 - 1 1 - 1 1 1 1 1 1 - 1 1 - 1 - 1 - 1 1 - 1 1 - 1 - 1 1 • Vbo 2 0 0 1 1 - 1 - 1 - 1 1 - 1 1 - 1 - 1 - 1 - 1 1 1 - 1 1 1 1 - 1 - 1 1 - 1 • IFTU Reliability Experiment Design • Endgame Accuracy (Fractional Factorial Method) *Note: Examples-only, not a complete list. 5

Polynomial Models for Sensitivity Analysis • Simple Additivity: – P.E. = b o + Σ b i X i ( i = 1,…, p factors) – X i = -1 or +1 (coded values for the factor with a span wide enough that should result in a lower P.E. and a higher P.E. if there is an effect) • Two-way Interactions: – P.E. = b o + Σ b i X i + Σ b ij X i X j (i ≠ j) many b ij terms Quadratic with two-way interactions: • – P.E. = b o + Σ b i X i + Σ b ij X i X j + Σ b ii X i2 – requires more than two levels for each factor 6

Factorial Designs • Varies many factors simultaneously, not the “change- one-variable-at-a-time” method. • Checks for interactions (non-additivity) among factors. • Shows the results over a wider variety of conditions. • Minimizes the number of computer simulation runs for collecting information. • Built-in replication for the factors to minimize variability due to random variables - usually no design point is replicated, all different points in the design matrix. 7

Screening Designs Full Factorial Design Fractional Factorial Design (R. A. Fisher - 1926) (Yates/Cochran/Finney - 1930’s) 2 3-1 = 4 points in each 1/2 fraction 2 3 = 8 points ⇓ • • • Use either the • • • • Green or Purple points • • • • • Huge efficiencies for large numbers of dimensions, such as 2 11- 4 , 2 47- 35 , or 2 121-113 . The “curse of dimensionality” is solved by fractional factorial designs. 8

Factorial Method: Full vs. Fractional )XOO )DFWRULDO 'HVLJQ )UDFWLRQDO )DFWRULDO 'HVLJQ P-K computer runs • Varies P factors at two levels • Requires 2 • Only hundreds to thousands of Requires 2 P computer runs • computer runs for 47 factors • If 47 factors 140 trillion • Assumptions: runs !! – Monotonicity (not Linearity) • Full information on: – Few higher order interactions – main effects are significant – two-way interactions • The terms of the model may not – three-way, four-way, …, be estimated separately, only the up to P- way interactions linear combinations of them Resolution Levels 9

Number of Runs Needed for Two-Level Fractional Factorials Resolution Levels Resolution 2 Main effects (b i ) confounded with themselves. Resolution 3 Main effects (b i ) not confounded with themselves, but with two-way effects (b ij ). Resolution 4 Main effects (b i ) not confounded with two-factor effects, but two-way effects (b ij ) confounded with themselves. Resolution 5 Main effects (b i ) and two-way effects (b ij ) not confounded with each other, but three-ways effects confounded with two-ways and four-ways with main effects. Number of Runs Required for a Number of Runs Required for a Resolution 5 Fractional Factorial Resolution 4 Fractional Factorial Number of Minimum Number Number of Minimum Number Factors of Runs Factors of Runs 1 2 1 2 2 4 2 4 8 = 2 3 3 8 = 2 3 3 – 4 16 = 2 4 4 – 5 16 = 2 4 5 – 8 32 = 2 5 6 – 7 32 = 2 5 9 – 16 64 = 2 6 8 64 = 2 6 17 – 32 128 = 2 7 9 – 11 128 = 2 7 33 – 64 256 = 2 8 12 – 17 256 = 2 8 65 – 128 512 = 2 9 18 – 22 512 = 2 9 129 – 256 1,024 = 2 10 23 – 31 2,048 = 2 11 32 – 40 4,096 = 2 12 41 – 54 10

Factors to be Screened Factors Factors (continued) 1 Threat RCS 25 PAC III Reaction Time 2 SBIR Prob of Detection 26 PAC III Pk 3 SBIR Network Delay 27 PAC III Vbo 4 SBIR Accuracy 28 AEGIS Time to Acquire Track 5 SBIR Time to Form Track 29 AEGIS Time to Discriminate 6 THAAD Time to Acquire Track 30 AEGIS Time to Commit 7 THAAD Time to Discriminate 31 AEGIS Time to Kill Assessment 8 THAAD Time to Commit 32 AEGIS Prob of Correct Discrimination 9 THAAD Time to Kill Assessment 33 AEGIS Prob of Kill Assessment 10 THAAD Prob of Correct Discrimination 34 AEGIS Launch Reliability 11 THAAD Prob of Kill Assessment 35 AEGIS Reaction Time 12 THAAD Launch Reliability 36 AEGIS Pk 13 THAAD Reaction Time 37 AEGIS Vbo 14 THAAD Pk 38 Network Delay 15 THAAD Vbo 39 Lower Tier Minimum Intercept Altitude 16 PATRIOT Time to Acquire Track 40 Upper Tier Minimum Intercept Altitude 17 PATRIOT Time to Discriminate 41 ABL Reaction Time 18 PATRIOT Time to Commit 42 ABL Beam Spread 19 PATRIOT Prob of Correct Discrimination 43 ABL Atmospheric Attenuation 20 PAC II Launch Reliability 44 THAAD Downtime 21 PAC II Reaction Time 45 PATRIOT Downtime 22 PAC II Pk 46 AEGIS Downtime 23 PAC II Vbo 47 ABL Downtime 24 PAC III Launch Reliability 11