. zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA 7 - PDF document

. zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA 7 zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA i zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA The Challenge A Cognitive Systems Engineering ~~ of DoD .Seeming mystique W&A Application of the VnlnerabilityLethality p o h Analysis Methodology to Simulation .Broad diflkulty experienced in Credibility Assessment implementhag policy - OW&A educational approachrs.ye* ~. very general .~ . , 7 ' > . _ .. O W & A zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA What is VV&A? W & A Defined is a method for Psseving simulation credibility application zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA O W & A reduces the r i s k s sssoeiated with simulation development and . B y verifying and validating the simulation, you gain confidence that it is the right tool to help you in your m m #Hypothesis: zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA ypothesis and Assumptions Thrt undcnlanding Ihe VV&A p n r a i c m be enhanced by luinba melhodology with which the audience is drrady familiar #Assumption L : The methodolow must be widely undrntaod and uwd within a spechie DoD community #Assumption1 I t m ~ ~ t b t p ~ u i b l e c o ~ d s p t Ihr melhdolog Io the VV&A problem I

Aore Definitions .Space- the set zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA information zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA .Level -the set zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA .Mapping zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA - Methodology Terminology Levels ofthe V n Methodolow ofdl combinations offactors that Level (-1) Weapon detection could result in a given system state after interaction zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA Level 0 lnltid t b r u t conditions of acrewary to Level 1 ulreat/target conditions describe the state ofthesystem Initial - Ihe types of information that allow Level 2 Danuge to taget .Iter i n t c d o n movement to another level Rmdnlng platform capability Level3 &er interaction Remaining pbtform utillty Level 4 n I support zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA Nelson's Application Required Information .Examine information requirements that 1 . Ukely combat renarios vn ana~yses 2. Scenario characteristies .Identify required information first 3 . Mission succen criteria 4. Auarment ofspcdtic capabilities ODctcrmine what required information available through txisting data soumes 5. Levels ofspecific upabilities 6 . .Determine whether the available information Subsystems to prwidc capabilities is also reliable 7. Critical components .Define data voids where required informalion 8. ukely dnmage to the system b either unavailable or unreliable 9. Expected vulnerabilities I . m 2

t zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA t@f zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA vulnerability of a simulation in its environment zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA Glasow's Application Vulnerabilities of Simulations .Simulations are subject to technical and political .The V/L methodology is a method for assessing influences that may degrade the intended the vulnerability of a weapon system in its capability or functionality of the simulation environment poorly designed simulation may result in A . +HOWEVER, zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA .The W & A process is a method for assessing the inaccurate results, negative training, or failure to suc~gsfully accomplish a mission A . n inaccurate simulation could result in program cancellation A WWPONSYSTEMI.~NOTASlMUUTlON .Simulation development may be cancelled if th ! ' . simulation fails to demonstrate significant enhancements to operational systems or cost effectiveness :;yc .Axiom zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA preceding level. zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA 1: Mappings from one level to the next are noninvertible. .Axiom 2: A level can be described with varying degrees of granularity, but the granularity at one level cannot exceed the granularity of its 9 - g ? ! .Axiom 3: Metrics must be measurable and objective. L Likely comha zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA Findings Common to Both the V/L Similarities in Information Requirements Methodology and the VV&A Process rrcnarins 1 . 1 Likcl? . simulation application .Capabilities are application-dependent 2 . Scmario rhuracterirticr 2. Simulation funnionalitics .Users, strategists, and analysts are preferred 3. hfivsion suwess criteria 3. Simulalian acccptahility criteria d. Airarment olspetific 1 . Vcritication and validation sources for data regarding performance cnpahilitin ( V e V ) critcria requirements 5. Lercb oriprrific capahilitia s . nmrhal,j, lor vev Tritelip 6. Suhryrtcms 10 prmidc 6. Simulation moduln nndd to .Tailor the effort on the rrpahilitia mcaJurr simulation pcdomanrc 7. Critical comjroncntq most critical aspects 7 . Criticnl algorithms and modules 10 the v s c m Lihl? d a m w 8. Likely rhangw 10 the .Learning and adaptation 9. Eipccted \ulncrahilitim 9 Eipcctrd . crrdihility is the logical next step 3

understanding of zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA .Political and technical changes zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA Conclusions .Use of a familiar, accepted methodology is a valuable and viable approach for enhancing intended use zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA within a specific W & A community .The methodology emphasizes simulation requirements and likely applications to simulation development are proactively anticipated .The potential impact of change is evaluated vis-&vis the simulation’s ability to meet the 4

1 zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA CREDIBILITY ASSESSMENT zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA A COGNITIVE SYSTEMS ENGINEERING APPLICATION OF THE VULNERABILITY/LETHALITY ANALYSIS METHODOLOGY TO SIMULATION Priscilla A. Glasow The MITRE Corporation 1820 Dolley Madison Boulevard (W626) McLean, Virginia 22 102 703-883-693 1 pglasow@mitre.org PURPOSE OF STUDY This paper examines the application of a widely-used methodology to determine the information requirements to support the Department of Defense’s (DoD) verification, validation, and accreditation (VV&A) process. VV&A of models and simulations is required by policy, Validation is the process of determining the degree to which a model is zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA specifically, DoD Instruction 5000.6 1. More importantly, VV&A simply makes sense. Rational, ethical analysts use analytical tools, such as simulation, in which they have confidence. VV&A simply assesses the credibility of the simulation tools that we use so that we can have confidence in their application and in their results. For the purposes of this paper, the DoD definitions for VV&A are used: Verification is the process of determining that a model accurately represents the developer’s conceptual description and specifications. accurate an representation of the real world. The intended use of the model determines the degree of accuracy required for validation. Accreditation is the official certification that a model or simulation is acceptable to use for a specific application. The current challenge in implementing the spirit and intent of DoD’s policy for VV&A is in overcoming a deeply embedded mystique that has arisen about this process. Despite years of informally ensuring the credibility of models and simulations, many organizations within DoD have been unable to grasp the relatively simple philosophy of VV&A or effectively implement VV&A guidelines in practice. The lack of a common basis for understanding VV&A is one of the many reasons that have been cited to explain this problem. RATIONALE FOR THE SELECTED METHODOLOGY Background This paper hypothesizes that understanding the VV&A process can be enhanced by using a methodology with which the target audience is already familiar. Two assumptions are made. First, the methodology selected must be widely understood and used within a specific DoD community. Second, it must be possible to adapt the methodology to the VV&A problem. The methodology chosen for this study was the VulnerabilityLethality analysis methodology and its associated taxonomy. Two presentations given at the 15‘h International Symposium on Military Operational Research served as the genesis for this paper. First, Deitz (1998) presented his findings from recent applications of the VulnerabilityAethality methodology. A second