OFT Degree of Networking: ASDC3I Network The extent to which force entities are interconnected Attribute Metrics Reach Percent of nodes that can communicate in desired access modes, information formats, and applications Quality of Service Vector of performance metrics, including average bandwidth provided (available and bottleneck), packet delay, delay jitter, and data loss Network Assurance Categorical rating from “highly secure” to “not secure” (estimated from assessment of network’s installed security software, hardware, and usage policies) Network Agility See next slide EBR RAND 8/3/2004 Slide 15
OFT Degree of Networking: ASDC3I Network Agility Attribute Metrics Robustness Number of differing conditions/environments over which network is capable of operating at a given level of effectiveness (baseline level determined by SME, simulation, analysis, empirical analysis, etc.) Effectiveness of network across varying levels of attack/degradation (baseline level determined by SME, simulation, analysis, empirical analysis, etc.) Number of tasks/missions which the network is capable of operating at a given level of effectiveness (baseline level determined by SME, simulation, analysis, empirical analysis, etc.) Responsiveness The timeliness of the response to an environmental change (baseline level determined by SME, simulation, analysis, empirical analysis, etc.) Flexibility Number of options for responding to an environmental change Compatibility of different responses (0=not compatible, 1=fully compatible; determined by SME, simulation, analysis, empirical analysis, etc.) Innovativeness Number of novel responses developed and implemented (baseline determined by SME, simulation, analysis, empirical analysis, etc .) Adaptiveness Number and timeliness of changes to network structure and processes (baseline determined by SME, simulation, analysis, empirical analysis, etc.) EBR RAND 8/3/2004 Slide 16
OFT NCW Conceptual Framework: ASDC3I Summary of Attributes (2) Degree of Information “Share-ability” Ease of Use Quantity of Posted Info Quantity of Retrievable Info Quality of Interactions Depth Quantity Quality Degree of Shared Information Quality of Individual Information Breadth Objective Measures Fitness for Use Reach Selectivity Objective Measures Intensity Completeness Extent Correctness Continuity Latency Quality Synchronicity Mode Accuracy Consistency Completeness Agility Correctness Robustness Flexible Relevance Accuracy Currency Response Consistency Innovative Adaptability Timeliness Relevance Precision Currency Individual Characteristics Risk Prop Competence Timeliness Precision Trust Quality of Individual Sensemaking: Awareness Org. Ident Confidence Degree of Shared Sensemaking: Shared Awareness Organizational Characteristics Fitness for Use Objective Measures Objective Measures Completeness Risk Prop Competence Extent Correctness Trust Confidence Quality Accuracy Size Hardness Correctness Consistency Completeness Diversity Permanence Relevance Autonomy Structure Consistency Accuracy Currency Interdepend Organizational & Individual Timeliness Currency Timeliness Behavior Precision Cooperation Efficiency Synchronization Precision Uncertainty Uncertainty EBR Engagement T vs. T RAND 8/3/2004 Slide 17
Quality of Interactions: OFT ASDC3I Dimensions and Attributes The focus of interaction: share information, develop and share awareness, develop and share understandings, make decisions Quality of Interactions • Depth • Breadth • Intensity • Agility Organizational and Organizational Individual Individual Behaviors Characteristics Characteristics •Cooperation •Risk Propensity •Efficiency •Risk Propensity •Competence •Synchronization •Competence •Trust •Engagement •Trust •Organizational Identification •Team vs. Task Balance •Confidence •Confidence • More .. EBR RAND 8/3/2004 Slide 18
Quality of Interactions OFT ASDC3I Top Level Attributes Attribute Definition Depth Measures that describe the nature of the substance of interactions Quantity The quantity of information, awareness, understandings, and/or decisions that are the focus of interactions Quality The quality of information, awareness, understandings, and/or decisions that are the focus of the interactions Breadth Measures that describe the force entities that interact Reach The number of members that participate in the interactions Selectivity The ability to reach a selected sub-set Intensity Measures that describe the pace and completeness of interactions Continuity The persistence of the exchange among members (continuous to episodic) Synchronicity Type of interaction: synchronous or asynchronous in time and space Mode Degree to which all senses are involved (ranges from face to face with data + voice to voice or data only) Latency The time lag of interactions Agility Robustness, Flexibility, Responsiveness, Innovativeness, and Adaptability EBR RAND 8/3/2004 Slide 19
OFT ASDC3I Degree of Shared Sensemaking • Shared Awareness - Those aspects of individual views of the battle space that are shared across two or more force entities/organizational members • Shared Understanding - Those recognitions, including patterns, cause and effect relationships, dynamic futures, and opportunities and risks, that are shared across two or more force entities/organizational members EBR RAND 8/3/2004 Slide 20
OFT Degree of Shared Sensemaking: ASDC3I Shared Awareness Attribute Definition Objective Measures Measures quality in reference to criteria that are independent of the situation Extent Proportion of awareness in common across force entities, within and across communities of interest (CoI) Proportion of force entities that share a given awareness Correctness Extent to which shared awareness is consistent with ground truth Consistency Extent to which shared awareness is consistent within and across CoI Currency Time lag of shared awareness Precision Level of granularity of shared awareness Quality Measures quality in reference to criteria that are determined by the situation Completeness Extent to which relevant shared awareness is obtained Accuracy Appropriateness of precision of shared awareness for a particular use Relevance Proportion of shared awareness obtained that is related to task at hand Timeliness Extent to which currency of shared awareness is suitable to its use Uncertainty Subjective assessment of confidence in shared awareness EBR RAND 8/3/2004 Slide 21
OFT NCW Conceptual Framework: ASDC3I Summary of Attributes (3) Quality of Individual Sensemaking: Awareness Degree of Shared Sensemaking: Shared Awareness Quality of Interactions Fitness for Use Objective Measures Extent Objective Measures Quantity Quality Completeness Depth Breadth Quality Correctness Accuracy Correctness Completeness Reach Selectivity Consistency Intensity Consistency Relevance Accuracy Continuity Latency Currency Currency Timeliness Timeliness Synchronicity Mode Precision Precision Uncertainty Uncertainty Agility Robustness Flexible Degree of Shared Sensemaking: Shared Understanding Objective Measures Quality of Individual Sensemaking: Understanding Response Fitness for Use Innovative Adaptability Extent Objective Measures Completeness Quality Individual Characteristics Correctness Correctness Completeness Risk Prop Competence Accuracy Consistency Accuracy Consistency Trust Relevance Org. Ident Confidence Currency Timeliness Currency Timeliness Precision Organizational Characteristics Uncertainty Precision Uncertainty Risk Prop Competence Trust Confidence Quality of Collaborative Decisions Quality of Individual Decisions Fitness for Use Fitness for Use Size Hardness Appropriateness Objective Measures Appropriateness Agility Agility Objective Measures Diversity Permanence Extent Completeness Completeness Consistency Robustness Robustness Autonomy Structure Consistency Accuracy Accuracy Interdepend Currency Flexibility Flexibility Currency Relevance Relevance Organizational & Individual Precision Responsiveness Responsiveness Behavior Precision Timeliness Timeliness Innovativeness Innovativeness Cooperation Efficiency Uncertainty Uncertainty Synchronization Adaptability Adaptability Mode of D. M. Mode of D. M. Risk Propensity Risk Propensity Engage T vs. T EBR RAND 8/3/2004 Slide 22
OFT Quality of Collaborative Decisions I ASDC3I Attribute Definition Objective Measures Measures quality in reference to criteria that are independent of the situation Extent Proportion of force entities that reach a collaborative decision Consistency Extent to which decisions are in agreement across force entities, within and across CoI Currency Time lag of decisions Precision Level of granularity of decisions Fitness for Use Measures Measures quality in reference to criteria that are determined by the situation Appropriateness Extent to which decisions are consistent with existing shared understanding, command intent and shared team values Completeness Extent to which relevant decisions encompass the necessary: •Depth: range of actions and contingencies included •Breadth: range of force elements included •Time: range of time horizons included Accuracy Appropriateness of precision of decisions for a particular use Relevance Proportion decisions that are important to the accomplishment of the task at hand Timeliness Extent to which currency of decision making is suitable to its use Uncertainty Inter-subjective assessment of confidence in decisions Risk Propensity Extent of risk aversion Mode of Decision Making Type of collaborative decision making structure utilized (authoritative decision making, consensus building, majority rule, etc.) EBR RAND 8/3/2004 Slide 23
OFT ASDC3I Quality of Collaborative Decisions II Attribute Definition Agility Robustness Degree to which collaborative decision is dominant across a range of situations and degradation conditions Flexibility Degree to which collaborative decision allows force entities to maintain flexibility (i.e., incorporates multiple ways of succeeding) Responsiveness Degree to which collaborative decision is relevant and timely Innovativeness Degree to which collaborative decision reflects novel ways to perform known tasks and/or develops new ways of doing novel tasks Adaptability Degree to which collaborative decision facilitates force entities’ ability to alter the decision, decision making participants and/or decision making process and implement appropriate modifications EBR RAND 8/3/2004 Slide 24
OFT NCW Conceptual Framework: ASDC3I Summary of Attributes (4) Quality of Individual Decisions Quality of Collaborative Decisions Fitness for Use Fitness for Use Objective Measures Agility Objective Measures Agility Appropriateness Appropriateness Robustness Extent Robustness Completeness Completeness Consistency Consistency Quality Accuracy Flexibility Accuracy Flexibility of Interactions Currency Currency Relevance Relevance Responsiveness Responsiveness Precision Precision Timeliness Timeliness Innovativeness Innovativeness Uncertainty Uncertainty Adaptability Adaptability Mode of D. M. Risk Propensity Mode of D. M. Risk Propensity Degree of Decision/ Plan Synchronization Synchronized Decisions/Plans Degree of Actions/ Entities Synchronized Synchronized Actions Synchronized Entities Operating Environments Degree of Effectiveness/ Agility Achievement of Objectives Agility Timeliness Efficiency EBR RAND 8/3/2004 Slide 25
OFT Degree of Decisions and ASDC3I Actions Synchronized Degree of Decisions / Plans Synchronized • Synchronized Decisions/Plans: Proportion of decisions/plans that are conflicted, de-conflicted or synergistic Degree of Actions / Entities Synchronized • Synchronized Actions: Proportion of actions that are conflicted, de-conflicted or synergistic • Synchronized Entities: Proportion of force entities whose positions are conflicted, de-conflicted, or synergistic EBR RAND 8/3/2004 Slide 26
OFT ASDC3I Degree of Effectiveness / Agility Attribute Definition Achievement of Degree to which Military/Strategic/ Political/ Social/ Economic/ Objectives Diplomatic objectives were achieved Agility The degree to which force entities were robust, flexible, responsive, innovative, and adaptable Time Time required to achieve objective Efficiency Total cost of achieving objective EBR RAND 8/3/2004 Slide 27
Framework: Attributes and Metrics OFT ASDC3I Information Value Added Force C2 Effectors Sources Services Degree of Networking Quality of Organic Information Degree of Information “Share-ability” Quality of Individual Information Degree of Shared Information Quality Degree of Shared Sensemaking Quality of Individual Sensemaking of Awareness Shared Awareness Inter- Understanding Shared Understanding actions Quality of Individual Decisions Quality of Collaborative Decisions Parts of framework Degree of Decision/ Plan Synchronization for which attributes and metrics have been developed Degree of Actions/ Entities Synchronized Parts of framework for which attributes have been developed Operating Environments Degree of Effectiveness/ Agility EBR RAND 8/3/2004 Slide 28
OFT ASDC3I Some Issues (1) • Social Domain – Is this really a domain? If so, what is its relationship to the cognitive domain? • Survivability – Are vulnerability and potential degradation of networked forces adequately accounted for in the framework? • Synchronization – Should this measure explicitly account for asynchronization? • Coherence – Should this be a separate measure? If so, how does it relate to synchronization? • Force Cohesion – Is this an important indicator of mission success? How does it relate to degree of interaction? EBR RAND 8/3/2004 Slide 29
OFT ASDC3I Some Issues (2) • Quality of Interactions – Are some of the attributes exogenous variables? • Integration – Should this be an explicit measure? – How does it relate to degree of interaction? • Agility – Is this measure adequately represented in the framework? – Should it be more systemic? • Mission Capability Packages – Should the relationship between exogenous variables and DOTML-PF be more explicit? • Operating Environment – Is its relationship to other measures proper? – What are the appropriate attributes? EBR RAND 8/3/2004 Slide 30
OFT ASDC3I Agenda • Informing Transformation • The NCW Framework Initiative • The NCW Framework • Elements of the Force • NCW Measures, Attributes and Metrics • Case Study: Air-to-Air Combat • Summary and Next Steps EBR RAND 8/3/2004 Slide 31
OFT ASDC3I The JTIDS Operational Special Project Results from 12,000 sorties in Air-to-Air Combat • Conditions -AWACS with fighter aircraft -Range from 2 on 4 aircraft up to 8 on 16 aircraft missions -Day and night engagements -Voice only vs. voice + Link 16 • Results (Kill Ratio, X:1) Voice Only Voice + Link 16 Day 3.10 8.11 Night 3.62 9.40 EBR RAND 8/3/2004 Slide 32
Air-to-Air Scenario Exemplar OFT ASDC3I Four-on-Four Engagement Red 1, 2 Blue11, 12 Red 3, 4 AWACS Blue13, 14 EBR RAND 8/3/2004 Slide 33
Framework: Data Available for Case Study OFT ASDC3I Information Value Added Force C2 Effectors Sources Services Degree of Networking Quality of Organic Information Degree of Information “Share-ability” Quality of Individual Information Degree of Shared Information Quality Degree of Shared Sensemaking Quality of Individual Sensemaking of Awareness Shared Awareness Inter- Understanding Shared Understanding actions Quality of Individual Decisions Quality of Collaborative Decisions Degree of Decision/ Plan Synchronization Parts of Framework Degree of Actions/ Entities Synchronized for which reasonable data is available Parts of Framework Operating Environments for which data is currently unavailable Degree of Effectiveness/ Agility EBR RAND 8/3/2004 Slide 34
OFT ASDC3I Force Mission Voice Only Voice + Link 16 Capability (MCP #1) (MCP #2) Packages Elements AWACS Voice Network Data +Network Air Craft (Network, Nodes) Information Value added Roles C2 Effectors Sources Services • Detect/ •Fuse data •Assign aircraft to •Kill Targets Functions ID targets •ID info targets •Distribute Info. •Coordinate engagements Embedded in • Coverage • Capability • Target Destruction Relevant the NCW • Persistence • Capacity Attributes conceptual • Performance • Quality of Service framework EBR RAND 8/3/2004 Slide 35
OFT Influence of the Force on Quality of ASDC3I Organic Information Force Information Sources Value Added Services C2 Effectors Quality of Organic Information Degree of Networking Fitness for Use Net Ready Nodes Objective Measures Network Completeness Capacity Correctness Reach Accuracy Consistency Connectivity Quality of Service Relevance Currency P&R Capability Support Network Assurance Timeliness Precision Collaboration Support Network Agility Node Assurance Quality of Individual Information Fitness for Use Degree of Information Objective Measures Completeness “Share-ability” Correctness Quantity of Posted Info Accuracy Quality Consistency of Quantity of Retrievable Info Relevance Interactions Currency Ease of Use Timeliness Precision EBR RAND 8/3/2004 Slide 36
OFT ASDC3I Computing Quality of Organic Information Force Information Sources Blue11 Blue12 Blue13 Blue14 AWACs Sensor coverage - Field of regard AMTI radar AMTI radar AMTI radar AMTI radar AMTI radar - Instantaneous field of view IFFN IFFN IFFN IFFN IFFN - Area coverage rate NCTR#1 NCTR#1 NCTR#1 NCTR#1 NCTR#1 - Revisit rate Etc. Etc. Etc. Etc. Etc. (e.g. AWACS 10 sec scan) - Range Probability of detection - Fn of RCS Exogenous variables - Fn of doppler • Environmental Probability of false alarm Quality of Organic Information conditions Probability of classification Fitness for Use • Doctrine Objective Measures Sighting location error Completeness Sighting velocity error Correctness Radar processor track precision Accuracy Consistency Currency Relevance f(…) Precision Timeliness EBR RAND 8/3/2004 Slide 37
Quality of Organic Information: OFT ASDC3I Threat Tracks Blue11, 12 1 0.75 0.5 0.25 0 T T T T Q Q Q Q 1 2 3 4 1 0.75 0.5 0.25 0.25 0 T T T T Blue13, 14 Q Q Q Q 1 2 3 4 0 T T T T Q Q Q Q 1 2 3 4 T T Q 3 Q 1 Correctness: Location Completeness: Detection T T Q 4 Q 2 Correctness: ID Correctness: Velocity EBR RAND 8/3/2004 Slide 38
OFT Comparing MCPs Using Summary Metrics ASDC3I Quality of Organic Information Quality of Individual Information Degree of Degree of Shared Information Information “Share-ability” Quality of Degree of Individual Networking * 0.28 Sensemaking Quality of 1.0 Quality of Individual Organic Info Decisions Kill Ratio (Synchronization / Effectiveness) Voice Overall average over information quality * * Voice + Link 16 dimensions and package members EBR RAND 8/3/2004 Slide 39
Comparing MCPs Using Summary Metrics OFT ASDC3I Synchronization and Effectiveness Quality of Individual Information Degree of Degree of Shared 0.91 Information Information “Share-ability” 1.0 1.0 0.4 0.22 0.08 0.5 0.91 1.0 Quality of Degree of 0.45 Individual 1.0 Networking 0.28 0.45 Sensemaking 3.10:1 0.91 Quality of Quality of 8.11:1 Individual Organic Info Decisions Kill Ratio (Synchronization / Effectiveness ) Voice Overall average over information quality * * Voice + Link 16 dimensions and package members EBR RAND 8/3/2004 Slide 40
OFT ASDC3I Quality of Sensemaking & Decisions Relative Speed and Competitive Advantage Hypotheses: • Information sharing via Voice + Link 16 leads to less time necessary to gather critical information, which results in more time available for flight lead to develop sensemaking and make decisions • Information sharing via Voice + Link 16 leads to less time necessary for wingman to gather and monitor critical information, which results in opportunities for wingman to spend time sensemaking and making decisions B11 (Flight lead) Information Awareness Understanding Decisions Voice Only Info Awareness Understanding Decisions Link-16 +Voice B12 (Wingman) Information Awareness Voice Only Link-16 +Voice Info Awareness Understanding Decisions Time EBR RAND 8/3/2004 Slide 41
OFT Degree of Synchronization and Effectiveness ASDC3I Reported Tactical Improvements Enabled by Voice + Link 16 • Voice + Link 16 allows greatly increased information sharing, leading to nearly- comprehensive awareness and understanding of air-to-air battlespace • Greater understanding allows for use of four types of “high-awareness” tactics that lead to major increases in combat effectiveness 1. Increased numbers of engagements in 2. Employment of the wingman as the same time period combatant rather than defensive patroller Flight lead Wingman Time 3. Advance vectoring to engage red A/Cs 4. Employment of cooperative formations to from position of maximum advantage trap and destroy red A/Cs EBR RAND 8/3/2004 Slide 42
OFT Areas that Require Additional ASDC3I Attention for Air-to-Air Case Study • Data describing cognitive and social behavior – Quality of interactions – Sensemaking / Decision making • Impact of non-material changes in DOTML-PF – C2 concept (e.g., role of AWACs) – Changes in tactics, techniques and procedures • Impact of changes in force mix – A/C, sensor and weapon type • Effects of scaling number of A/C – Impact on net performance – Impact on mission effectiveness • Accounting for dynamics over time EBR RAND 8/3/2004 Slide 43
OFT ASDC3I Agenda • Informing Transformation • The NCW Framework Initiative • The NCW Framework • Elements of the Force • NCW Measures, Attributes and Metrics • Case Study: Air-to-Air Combat • Summary and Next Steps EBR RAND 8/3/2004 Slide 44
OFT Potential Next Steps ASDC3I “Getting NCW Theory and Metrics Right…” – Continue to refine and evolve the framework – Complete Air-to-Air case study • Obtain additional data and address remaining measures – Disseminate framework and obtain additional peer review “…And Applied Enterprise-Wide” – Engage potential users of framework to establish new opportunities for application – Develop methodologies for applying framework in support of transformation – Conduct broad range of case studies with key partners Establish Board of Directors to shape priorities and ensure quality EBR RAND 8/3/2004 Slide 45
OFT ASDC3I Closing Thoughts • We are making progress in developing/applying the framework - Well into definition of second generation framework - However, significant issues remain • There is growing interest in applying the framework - Wide range of potential applications - Numerous opportunities for collaboration • Important to keep up momentum - Refine/extend framework - Identify and enable key applications • Broad community-wide participation is critical EBR RAND 8/3/2004 Slide 46
OFT Computing Quantity of Posted Info: ASDC3I Track Info over Voice Only Force Information Sources Value Added Services C2 Effectors Degree of Networking •AWACS, F-15s can transmit Net Ready Nodes over radio Network Capacity Reach •One military vocoder channel Connectivity Quality of Service •Assumed to be 100% in a P&R Capability Support Network Assurance benign environment Collaboration Support •No adjustments (static voice Network Agility Exogenous variables: broadcasting network) •Nodes: AWACS, F-15s Node Assurance •Types: 4 red tracks, 5 blue tracks, etc. Degree of Information •CONOPS, coding schemes, “Share-ability” governing how to speak track info f(…) Quantity of Posted Info •Red tracks have priorities, but two strike packages must know each other’s Quantity of Retrievable Info • In this scenario, quantity of posted positions info equals quantity of retrievable info, •Track info “expires” after except for probability of hearing voice Ease of Use ten seconds ! ! General ! ! Link 16 EBR RAND 8/3/2004 Slide 47
OFT Computing Quantity of Posted Info: ASDC3I Track Info over Link 16 Force Information Sources Value Added Services C2 Effectors Degree of Networking •AWACS, F-15s can transmit Net Ready Nodes track info over Link 16 Network Capacity Reach •One shared Link 16 network (capacity greatly exceeds Connectivity number of info items here) Quality of Service •Assumed to be 100% in a P&R Capability Support Network Assurance benign environment Collaboration Support •No adjustments (static wireless Network Agility Exogenous variables: network) •Nodes: AWACS, F-15s Node Assurance •Types: 4 red tracks, 5 blue tracks, etc. Degree of Information •CONOPS, coding schemes, “Share-ability” governing how to speak track info f(…) Quantity of Posted Info •Red tracks have priorities, but two strike packages must know each other’s Quantity of Retrievable Info • In this scenario, quantity of positions posted info over Link 16 equals •Track info “expires” after quantity of retrievable info Ease of Use ten seconds ! ! ! General ! Voice Only EBR RAND 8/3/2004 Slide 48
Computing Quantity of Posted Info: OFT ASDC3I Detailed Function for Posted Info Force Source/Sensors Value Added Info Processors C2 Effectors Exogenous variables: Degree of Networking •Whether nodes can transmit to •Number of nodes Net Ready Nodes network •File sizes and number of Network files Capacity Reach •Variables impacting how •Posting channel numbers, types, quickly nodes can transmit and bandwidth (for data links Connectivity pieces of information only) Quality of Service (CONOPS, coding schemes, etc.) •Prob of correct transmission P&R Capability Support •Policies determining Network Assurance priority for posting •Expiration age for each Collaboration Support •Adjustments to probability that type of info objects Network Agility QoS will be delivered •Maximum queue lengths Node Assurance f(…): Vector for number of info objects that can be posted, by object type Phase 1. For each type of info object do: Degree of Information • If nodes can post object, do 2. Else, Num(type) = 0 “Share-ability” • Use QoS parameters, network agility parameters, and exo variables to determine rate at which nodes can post info items of that type. Multiply this Quantity of Posted Info rate by probability of correct transmission, yielding theoretical transmission rate . Phase 2. Quantity of Retrievable Info • Use theoretical rates for each info type plus priority policies to determine what fractions of postings will be of each info type. Ease of Use • Multiply fractions of postings times theoretical rates times expiration age to get Num(type) for each info type. ! ! ! General ! EBR RAND 8/3/2004 Slide 49
Computing Quantity of Posted Info: OFT ASDC3I Detailed Function for Voice Only Network Force Source/Sensors Value Added Info Processors C2 Effectors Degree of Networking •AWACS, F-15s can transmit Exogenous variables: Net Ready Nodes over radio •Nodes: AWACS, F-15s Network •Types: 4 red tracks, 5 blue Capacity Reach tracks, etc. •CONOPS, coding schemes, •One military vocoder channel Connectivity governing how to speak Quality of Service track info •Red tracks have priorities, •Assumed to be 100% in a P&R Capability Support but two strike packages Network Assurance benign environment must know each other’s positions Collaboration Support •No adjustments (static voice •Track info “expires” after Network Agility f(…): Vector for number of info objects that broadcasting network) ten seconds can be retrieved, by object type Node Assurance • 70% of voice tracks are audible, so metric is 0.7*Quantity of Posted Info f(…): Vector for number of info objects that can be posted, by object type Phase 1. For each type of info object do: Degree of Information • AWACS, F-15s can post (do 2) “Share-ability” • Using AF CONOPS and coding standards, AC can transmit three tracks every ten seconds on a military coding voice channel. In this scenario, this rate is Quantity of Posted Info unaffected by network assurance considerations. Phase 2. • Using AF priority policies, on average each 10-sec interval has two red tracks Quantity of Retrievable Info and one blue track. • Since each track “lasts” for ten seconds, at most an average of two red tracks Ease of Use and one blue track can be posted at any time. ! ! ! ! Voice EBR RAND 8/3/2004 Slide 50
Computing Quantity of Posted Info: OFT ASDC3I Detailed Function for Link 16 Network Force Source/Sensors Value Added Info Processors C2 Effectors Degree of Networking •AWACS, F-15s can transmit Exogenous variables: Net Ready Nodes track info over Link 16 •Nodes: AWACS, F-15s Network •Types: 4 red tracks, 5 blue Capacity Reach tracks, etc. •One shared Link 16 network •CONOPS, coding schemes, (capacity greatly exceeds Connectivity governing how to speak number of info items here) Quality of Service track info •Red tracks have priorities, •Assumed to be 100% in a P&R Capability Support but two strike packages Network Assurance benign environment must know each other’s positions Collaboration Support •No adjustments (static wireless •Track info “expires” after Network Agility network) ten seconds Node Assurance F(…): Vector for number of info objects that can be posted, by object type Phase 1. For each type of info object, do: Degree of Information • AWACS, F-15s can post (do 2) “Share-ability” • Using Link 16 capacity and AF track coding standards, rate exceeds maximum number of tracks updated every second. Quantity of Posted Info Phase 2. • All tracks can be posted at least every second, so no priority policies apply. • Info on all tracks can be updated every second, so info on all nine tracks can Quantity of Retrievable Info be posted at any given time. Ease of Use ! ! Link 16 ! ! EBR RAND 8/3/2004 Slide 51
OFT Computing Correctness for Individual ASDC3I Information: Voice Only Quality of Organic Information Exogenous variables: •Organic info assumed to Fitness for Use Objective Measures •Track, capability, intent be correct, within known Completeness information, all in margins of error Correctness standard formats •No fusion performed Accuracy Consistency •Voice track messages only •Pilots will use organic audible 70% of the time information in preference Currency to radio-reported Relevance information •Military vocoder replicates •Pilots have sufficient Precision hearing errors (no Timeliness training to use radio correction7) f(…) Quality of Individual Information Fitness for Use Degree of Information Objective Measures Completeness “Share-ability” Correctness Quantity of Posted Info Accuracy Consistency Quantity of Retrievable Info Relevance Currency Ease of Use Timeliness Precision ! General ! ! ! Link 16 EBR RAND 8/3/2004 Slide 52
OFT Computing Correctness for Individual ASDC3I Information: Link 16 Exogenous variables: •Track, capability, intent Quality of Organic Information •Organic info assumed to information, all in be correct, within known Fitness for Use Objective Measures standard formats margins of error Completeness •Fusion consolidates blue Correctness tracks only •Pilots will use F-15 radar Accuracy Consistency •Assuming Link 16 reaches information in preference blue AC with no info to AWACS information degradation Currency •Pilots have sufficient Relevance training to use Link 16 display and radio Precision •Display screen assumed to Timeliness be error free within screen resolution f(…) Quality of Individual Information Fitness for Use Degree of Information Objective Measures Completeness “Share-ability” Correctness Quantity of Posted Info Accuracy Consistency Quantity of Retrievable Info Relevance Currency Ease of Use Timeliness Precision ! ! ! General ! Voice EBR RAND 8/3/2004 Slide 53
OFT Computing Correctness for Individual ASDC3I Information: Detailed Function Objective Measures Fitness for Use •Correctness of organic Correctness Completeness information Consistency Accuracy Quality of Exogenous variables: Organic •Types of information Currency Relevance Information •Fusion performed on the information, and quality Precision Timeliness of this process •Training to use retrieve and present information •Whether the info was retrieved in original form Quantity of Posted Info Degree of Information Quantity of Retrievable Info •Whether the presentation “Share-ability” of the info introduces errors Ease of Use F(…): Correctness of information object Objective Measures Fitness for Use • If info comes from an organic source, correctness = organic correctness Correctness Completeness • If info is from network, correctness is the original correctness “multiplied” by the probability the info was Consistency Accuracy Quality of retrieved and presented in original form. Individual • If info underwent fusion (esp. if taken from multiple Currency Relevance Information sources), correctness is “multiplied” by additional factor representing fusion effectiveness. Precision Timeliness ! ! ! General ! EBR RAND 8/3/2004 Slide 54
OFT Computing Correctness for Individual ASDC3I Information: Detailed Function, Voice Only Objective Measures Fitness for Use •Organic info assumed to Correctness Completeness be correct, within known margins of error Exogenous variables: Consistency Accuracy Quality of •Track, capability, intent Organic information, all in Currency Relevance Information standard formats •No fusion performed Precision Timeliness •Pilots will use organic •Assuming voice signal information in preference reaches blue AC with no info to radio-reported degradation, but has only 70% Quantity of Posted Info information Degree of chance of being audible •Pilots have sufficient Information training to use radio Quantity of Retrievable Info “Share-ability” •Military vocoder repeats Ease of Use auditory errors F(…): Correctness of information object Objective Measures Fitness for Use • Assumed correctness for each of blue AC’s own tracks • If info received from vocoder, approximately a 70% Correctness Completeness chance that message will have been heard correctly Fusion does not apply in this case. Consistency Accuracy Quality of Total: 100% of organic info objects are correct; only 70% of Individual voice-reported info objects are correct (others are Currency Relevance Information garbled to point of unusability) Precision Timeliness ! ! ! Voice ! EBR RAND 8/3/2004 Slide 55
OFT Computing Correctness for Individual ASDC3I Information: Detailed Function, Link 16 Objective Measures Fitness for Use •Organic info assumed to Correctness Completeness be correct, within known Exogenous variables: margins of error •Track, capability, intent Consistency Accuracy Quality of information, all in Organic standard formats Currency Relevance Information •Fusion consolidates blue tracks only Precision Timeliness •Pilots will use F-15 radar information in preference •Assuming Link 16 reaches to AWACS information blue AC with no info •Pilots have sufficient degradation Quantity of Posted Info Degree of training to use Link 16 Information display and radio Quantity of Retrievable Info •Display screen assumed to “Share-ability” be error free within screen Ease of Use resolution F(…): Correctness of information object Objective Measures Fitness for Use • Assumed correctness for each of blue AC’s own tracks • No errors introduced by Link 16 or info display (and Correctness Completeness pilots adequately trained to use display) Total: 100% of info objects are correct, whether organic or Consistency Accuracy Quality of shared Individual Currency Relevance Information Precision Timeliness ! ! Link 16 ! ! EBR RAND 8/3/2004 Slide 56
Computing Extent of Shared Information: OFT ASDC3I Detailed Function Whether Whether information can Degree of Information “Share-ability” Quality of Interactions sender can be physically Quantity of Posted Info communicate shared across Depth Quantity of Retrievable Info info with network Quantity Quality Ease of Use Breadth receiver Reach Selectivity Intensity Whether sender Degree of Shared Information Continuity Latency Objective Measures and receiver are Synchronicity Mode Extent: Proportion of force entities part of the same f(…) Agility that share information collaborative Robustness Flexible Quality group Response Matrix showing probabilities that particular information Completeness Innovative elements have been shared with particular users Adaptability Correctness For each element of the matrix, Pr(shared) is the product of: Individual Characteristics Accuracy • the probability that the info is retrievable; Risk Prop Competence Consistency • the probability the sender and receiver are part of the same Trust Relevance Currency collaborative group; Org. Ident Confidence • the probability the sender and receiver can communicate Number of communications Timeliness Organizational Characteristics Precision within the collaborative group; “hops” between sender and Risk Prop Competence • the probability the sender attempts to share the information receiver Degree of Shared Sensemaking: Shared Awareness Trust Confidence with the receiver; and Objective Measures Size Hardness • the probability the information is not degraded as a function Extent of the number of “hops” between sender and receiver. Diversity Permanence Quality Autonomy Structure Probability that sender Correctness Completeness Interdepend will attempt to share Organizational & Ind. Behavior information with Consistency Accuracy Cooperation Efficiency receivers Syn Currency Timeliness Deg. Of Eng T vs. T Precision Uncertainty ! ! ! General ! EBR RAND 8/3/2004 Slide 57
OFT Quality of Individual Sensemaking and ASDC3I Decision Making: Timeliness (Qualitative) Quality of Interactions Quality of Individual Sensemaking: Awareness Degree of Information “Share-ability” Fitness for Use Depth Quantity of Posted Info Objective Measures Completeness Quantity Quality Quantity of Retrievable Info Breadth Correctness Accuracy Reach Selectivity Ease of Use Consistency Intensity Relevance Continuity Latency Currency Timeliness Exogenous variables: Synchronicity Mode Precision Uncertainty training, experience, etc. Agility Robustness Flexible F(…): Timeliness of Sensemaking / Decision Making Response Quality of Individual Sensemaking: Understanding Fitness for Use Innovative f(…) Adaptability • If information is shared among all participants, less time is spent gathering Objective Measures Completeness and validating information, improving the timeliness of sensemaking and Individual Characteristics Correctness decision making Risk Prop Competence Accuracy Consistency • Real time interactions result in more efficient use of time, improving the Trust •Whether the focus of Relevance timeliness of sensemaking and decision making Org. Ident Confidence Currency interactions is on information Timeliness • Flexible command structures allow force members to make decisions with Organizational Characteristics gathering and validation or Precision fewer requirements, shortening decision-making times Uncertainty sensemaking/decision making Risk Prop Competence •Whether all relevant members Trust Confidence Quality of Individual Decisions are participating Fitness for Use Size Hardness •Whether the intensity of the Appropriateness Agility Objective Measures Diversity Permanence interactions matches the Completeness Robustness Autonomy Structure Consistency requirements of the mission Accuracy Interdepend Currency Flexibility Organizational & Ind. Behavior •Whether the command structure allows Relevance Precision Responsiveness Cooperation Efficiency for flexible roles and distributed decision Timeliness Synchronization Innovativeness making Engagement T vs. T Uncertainty Adaptability Mode of D. M. Risk Propensity ! ! ! General ! EBR RAND 8/3/2004 Slide 58
Quality of Individual Sensemaking and OFT ASDC3I Decision Making: Timeliness Voice Only (Qualitative) Quality of Interactions Quality of Individual Sensemaking: Awareness Degree of Information “Share-ability” Fitness for Use Depth Quantity of Posted Info Objective Measures Completeness Quantity Quality Quantity of Retrievable Info Breadth Correctness Accuracy Reach Selectivity Ease of Use Consistency Intensity Relevance Continuity Latency Currency Timeliness Exogenous variables: Synchronicity Mode Precision Uncertainty training, experience, etc. Agility Robustness Flexible F(…): Timeliness of Sensemaking / Decision Making Response Quality of Individual Sensemaking: Understanding Fitness for Use Innovative f(…) Adaptability • Participants spend most of their time gathering and validating information Objective Measures Completeness from AWACS and other blue AC radars Individual Characteristics Correctness • Voice communications adds delay over visual communications Risk Prop Competence Accuracy Consistency • Inflexible command structures require a variety of explicit checks and Trust •Whether the focus of Relevance permissions before engagement decisions can be made Org. Ident Confidence Currency interactions is on information Timeliness Organizational Characteristics gathering and validation or Precision Uncertainty sensemaking/decision making Risk Prop Competence •The number of participants Trust Confidence Quality of Individual Decisions •Whether the intensity of the Fitness for Use Size Hardness interactions matches the Appropriateness Agility Objective Measures Diversity Permanence requirements of the mission Completeness Robustness Autonomy Structure Consistency Accuracy Interdepend Currency Flexibility Organizational & Ind. Behavior •Whether the command structure allows Relevance Precision Responsiveness Cooperation Efficiency for flexible roles and distributed decision Timeliness Synchronization Innovativeness making Engagement T vs. T Uncertainty Adaptability Mode of D. M. Risk Propensity ! ! ! General ! EBR RAND 8/3/2004 Slide 59
Quality of Individual Sensemaking and OFT ASDC3I Decision Making: Timeliness Link 16 (Qualitative) Quality of Interactions Quality of Individual Sensemaking: Awareness Degree of Information “Share-ability” Fitness for Use Depth Quantity of Posted Info Objective Measures Completeness Quantity Quality Quantity of Retrievable Info Breadth Correctness Accuracy Reach Selectivity Ease of Use Consistency Intensity Relevance Continuity Latency Currency Timeliness Exogenous variables: Synchronicity Mode Precision Uncertainty training, experience, etc. Agility Robustness Flexible F(…): Timeliness of Sensemaking / Decision Making Response Quality of Individual Sensemaking: Understanding Fitness for Use Innovative f(…) Adaptability • Participants automatically receive all relevant information available from Objective Measures Completeness AWACS and other blue AC radars, so pilots incur no delays by Individual Characteristics Correctness communicating this information verbally Risk Prop Competence Accuracy Consistency • Near-real time visual information displays are much faster than voice Trust •Whether the focus of Relevance transmissions Org. Ident Confidence Currency interactions is on information Timeliness • Flexible command structures allow pilots to engage targets, and support Organizational Characteristics gathering and validation or Precision engaging pilots, directly. (Commanders only intervene when necessary.) Uncertainty sensemaking/decision making Risk Prop Competence •The number of participants Trust Confidence Quality of Individual Decisions •Whether the intensity of the Fitness for Use Size Hardness interactions matches the Appropriateness Agility Objective Measures Diversity Permanence requirements of the mission Completeness Robustness Autonomy Structure Consistency Accuracy Interdepend Currency Flexibility Organizational & Ind. Behavior •Whether the command structure allows Relevance Precision Responsiveness Cooperation Efficiency for flexible roles and distributed decision Timeliness Synchronization Innovativeness making Engagement T vs. T Uncertainty Adaptability Mode of D. M. Risk Propensity ! ! ! General ! EBR RAND 8/3/2004 Slide 60
OFT ASDC3I DEFINITIONS OF ATTRIBUTES EBR RAND 8/3/2004 Slide 61
OFT Quality of Organic Information ASDC3I Information gathered by individual sensors that is not shared and is unavailable to the network Attribute Summary Metrics Attribute Definition (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Extent to which information is consistent with ground truth Consistency Extent to which information is consistent with prior information Currency Age of information Precision Level of measurement detail of information item tuu Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Extent to which information relevant to ground truth is collected Accuracy Appropriateness of precision of information for a particular use Relevance Proportion of information collected that is related to task at hand Timeliness Extent to which currency of information is suitable to its use EBR RAND 8/3/2004 Slide 62
OFT ASDC3I Degree of Information “Share-ability” The degree to which information could be shared among force entities Attribute Summary Metrics Attribute Definition (Click Here) (Click Here) Quantity of Posted Extent to which collected information is posted Information Quantity of Proportion of nodes that can retrieve various sets of information. Determined Retrievable by the following: Information •Awareness of Information: Degree to which the existence of the information is advertised to force member •Access to Information: Degree to which access to information is controlled • Meta-data of Information: Degree to which information has labels describing what it is and how it may be used (facilitates indexing and searching) Ease of Use Degree to which presentation of information facilitates desired use EBR RAND 8/3/2004 Slide 63
OFT Quality of Individual Information ASDC3I Information gathered by individuals from the network and organic sources Attribute Summary Metrics Attribute Definition (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Extent to which information is consistent with ground truth Consistency Extent to which information is internally consistent with prior information/ awareness / understanding Currency Age of information Precision Level of measurement detail of information item tuu Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Extent to which information relevant to ground truth is obtained Accuracy Appropriateness of precision of information for a particular use Relevance Proportion of information retrieved that is related to task at hand Timeliness Extent to which currency of information is suitable to its use EBR RAND 8/3/2004 Slide 64
OFT ASDC3I Quality of Individual Sensemaking • Awareness - An individual’s holistic view of the battlespace that includes mission constraints, environment, time space relationships, the capabilities and intentions of red, blue, and neutral forces and an assessment of the associated uncertainties • Understanding - An individual’s recognition of patterns, cause and effect relationships, dynamic futures, and opportunities and risks EBR RAND 8/3/2004 Slide 65
OFT Quality of Individual Sensemaking: ASDC3I Awareness Metrics Attribute Summary Attribute Definition (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Extent to which awareness is consistent with ground truth Consistency Extent to which awareness is internally consistent with prior awareness Currency Time lag of awareness Precision Level of granularity of awareness tuu Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Extent to which relevant awareness is obtained Accuracy Appropriateness of precision of awareness for a particular use Relevance Proportion of awareness obtained that is related to task at hand Timeliness Extent to which currency of awareness is suitable to its use Uncertainty Subjective assessment of confidence in awareness EBR RAND 8/3/2004 Slide 66
OFT Quality of Individual Sensemaking: ASDC3I Understanding Attribute Summary Metrics Attribute Definition (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Extent to which understanding is consistent with ground truth Consistency Extent to which understanding is internally consistent with prior understanding Currency Time lag of understanding Precision Level of granularity of understanding Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Extent to which relevant understanding is obtained Accuracy Appropriateness of precision of understanding for a particular use Relevance Proportion of understanding obtained by force member that is related to task at hand Timeliness Extent to which currency of understanding is suitable to its use Uncertainty Subjective assessment of confidence in understanding EBR RAND 8/3/2004 Slide 67
OFT Quality of Individual Decisions I ASDC3I The extent to which an individual’s decisions build upon awareness and understanding Attribute Summary Attribute Definition (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Consistency Extent to which decisions are internally consistent with prior understanding and decisions Currency Time lag of decisions Precision Level of granularity of decisions Fitness for Use Measures Measures quality in reference to criteria that are determined by the situation Appropriateness Extent to which decisions are consistent with existing understanding, command intent and values Completeness Extent to which relevant decisions encompass the necessary: •Depth: range of actions and contingencies included •Breadth: range of force elements included •Time: range of time horizons included Accuracy Appropriateness of precision of decisions for a particular use Relevance Proportion of decisions that are significant to task at hand Timeliness Extent to which currency of decision making is suitable to its use Uncertainty Subjective assessment of confidence in decisions Risk Propensity Extent of risk aversion Mode of Decision Making Type of decision making process utilized (naturalistic, dominated, min-max, expected utility) EBR RAND 8/3/2004 Slide 68
OFT Quality of Individual Decisions II ASDC3I Attribute Summary Attribute Definition (Click Here) Agility Robustness Degree to which decision is dominant across a range of situations and degradation conditions Flexibility Degree to which decision allows force entities to maintain flexibility (i.e., incorporates multiple ways of succeeding) Responsiveness Degree to which decision is relevant and timely Innovativeness Degree to which decision reflects novel ways to perform known tasks and/or develops new ways of doing novel tasks Adaptability Degree to which decision facilitates force entities’ ability to alter the decision, decision making participants and/or decision making process and implement appropriate modifications EBR RAND 8/3/2004 Slide 69
OFT Quality of Interactions: ASDC3I Definitions and Explanations • Interactions involve force entities actively sharing information, and developing awareness, understanding and/or making decisions (developing plans) in a collaborative fashion while working together toward a common purpose • The focus of interactions: information sharing, developing and sharing awareness, developing and sharing understandings, making decisions • Attributes of interactions – Depth, breadth, intensity, agility • Contributing attributes – Individual Characteristics: risk propensity, competence, trust, organizational identification, confidence – Organizational Characteristics: risk propensity, competence, trust, confidence, size, hardness, diversity, permanence, autonomy, structure, interdependence – Organizational & Individual Behaviors: cooperation, efficiency synchronization, engagement, team vs. task balance EBR RAND 8/3/2004 Slide 70
OFT ASDC3I Quality of Interactions Contributing Attributes: Individual Characteristics Attribute Summary Attributes Definitions (Click Here) Risk Propensity Extent of risk aversion Competence Level of knowledge, skills, abilities, and attitudes (KSAAs) Trust Extent to which individual is willing to rely on other members Organizational Extent to which individual’s identities align with organizational identities Identification Confidence Degree of individual’s expectation that other members are reliable EBR RAND 8/3/2004 Slide 71
Quality of Interactions OFT ASDC3I Contributing Attributes: Organizational Characteristics Attribute Summary Attributes Definitions (Click Here) Risk Propensity Extent of risk aversion Competence Distribution of members knowledge, skills, abilities and attitudes (KSAAs) Trust Extent to which members are willing to rely on one another Confidence Extent to which members have expectations of the reliability of the organization Size Number of team members involved Hardness Degree to which team members have interacted in the past on the same task Diversity Degree to which team members are heterogeneous or homogeneous across exogenous variables: experience, age, gender, etc. Permanence Expected duration of organization Autonomy Extent to which organization is externally or self directed Structure Distribution of peer and authority relationships • Layers of authority • Functional Differentiation • Connectedness within and across layers • Directness of connections Interdependence Extent to which members depend on one another for resources (materials, KSAAs, etc.) EBR RAND 8/3/2004 Slide 72
OFT ASDC3I Quality of Interactions Contributing Attributes: Organizational and Individual Behaviors Attribute Summary Attributes Definitions (Click Here) Cooperation Extent to which member(s) are willing and able to work together Efficiency Extent to which members utilize one another’s resources so as to minimize costs and maximize benefits Synchronization Extent to which organization is conflicted, deconflicted, or synergistic Engagement Extent to which all members actively and continuously participate Team vs. Task Extent to which efforts are directed to organizational issues vs. relating to the Balance objective EBR RAND 8/3/2004 Slide 73
OFT Degree of Shared Information ASDC3I Attribute Summary Attribute Definition (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Extent Proportion of information in common across force entities, within and across communities of interest (CoI) Proportion of force entities that share information item Correctness Extent to which shared information is consistent with ground truth Consistency Extent to which shared information is consistent within and across CoI Currency Age of shared information Precision Level of measurement detail of shared information item Quality Measures quality in reference to criteria that are determined by the situation Completeness Extent to which shared information relevant to ground truth is obtained Accuracy Appropriateness of precision of shared information for a particular use Relevance Proportion of shared information retrieved that is related to task at hand Timeliness Extent to which currency of shared information is suitable to its use EBR RAND 8/3/2004 Slide 74
OFT Degree of Shared Sensemaking: ASDC3I Shared Understanding Attribute Summary Attribute Definition (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Extent Proportion of understanding in common across force entities, within and across communities of interest (CoI) Proportion of force entities that share a given understanding Correctness Extent to which shared understanding is consistent with ground truth Consistency Extent to which shared understanding is consistent within and across CoI Currency Time lag of shared understanding Precision Level of granularity of shared understanding Quality Measures quality in reference to criteria that are determined by the situation Completeness Extent to which relevant shared understanding is obtained Accuracy Appropriateness of precision of shared understanding for a particular use Relevance Proportion of shared understanding that is related to task at hand Timeliness Extent to which currency of shared understanding is suitable to its use Uncertainty Subjective assessment of confidence in shared understanding EBR RAND 8/3/2004 Slide 75
OFT ASDC3I DEFINITION OF METRICS EBR RAND 8/3/2004 Slide 76
OFT Quality of Organic Information ASDC3I Information gathered by individual sensors that is not shared and is unavailable to the network Attribute Summary Definitions Attribute Metrics (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Correspondence with ground truth-correlation coefficient (0= no correspondence with ground truth, 1= full correspondence with ground truth). Data matrix comprised of relevant information items estimates (for instance: detection, ID, velocity, location, heading, etc.) Consistency Degree of ‘deviation’ from previous information Currency Age of information Precision Level of measurement detail of information item Fitness for Use Measures quality in reference to criteria that are determined by the Measures situation Completeness Percentage of ground truth relevant and needed information collected Accuracy Degree to which precision matches what is needed (0=no match, 10=high degree of matching between precision level needed and available) Relevance Proportion of information collected that is related to task at hand Timeliness Degree to which currency matches what is needed (0=no match, 10=high degree of matching between currency level needed and available) EBR RAND 8/3/2004 Slide 77
OFT Degree of Information “Share-ability” ASDC3I The degree to which information could be shared among force entities Attribute Summary Definitions Attribute Metrics (Click Here) (Click Here) Quantity of Posted Percent of collected information posted Information Quantity of Percentage of nodes that can retrieve various sets of information. Retrievable Information Ease of Use Degree to which information is easy to use (0=low degree of ease of use, 10=high degree of ease of use) EBR RAND 8/3/2004 Slide 78
OFT Quality of Individual Information ASDC3I Information gathered by individuals from the network and organic sources Attribute Summary Definitions Attribute Metrics (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Correspondence with ground truth-correlation coefficient (0= no correspondence with ground truth, 1= full correspondence with ground truth). Data matrix comprised of relevant information items estimates (for instance: detection, ID, velocity, location, etc.) Consistency Degree of ‘deviation’ from previous information Currency Age of information Precision Level of measurement detail of information item Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Percentage of ground truth relevant and needed information Accuracy Degree to which precision matches what is needed (0=no match, 10=high degree of matching between precision) Relevance Proportion of information that is related to task at hand Timeliness Degree to which currency matches what is needed (0=no match, 10=high degree of matching between currency level needed and available) EBR RAND 8/3/2004 Slide 79
OFT Quality of Individual Sensemaking: ASDC3I Awareness Attribute Summary Definitions Attribute Metrics (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Correspondence with ground truth-correlation coefficient (0= no convergence, 1=full convergence between individual’s awareness and ground truth) Consistency Degree of ‘deviation’ from awareness gained from previous time period Currency Time lag of awareness Precision Level of granularity of awareness Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Percentage of ground truth picture included in awareness Accuracy Degree to which precision matches what is needed (0=no match, 10=high degree of matching between precision level needed and available) Relevance Proportion of awareness that is related to task at hand Timeliness Degree to which currency matches what is needed (0=no match, 10=high degree of matching between currency level needed and available) Uncertainty Confidence level (0% =uncertain, 100%= certain) or confidence interval (95%, 90%, etc.) of awareness EBR RAND 8/3/2004 Slide 80
OFT Quality of Individual Sensemaking: ASDC3I Understanding Attribute Summary Definitions Attribute Metrics (Click Here) (Click Here) Objective Measures Measures quality in reference to criteria that are independent of the situation Correctness Correspondence with ground truth-correlation coefficient (0= no convergence, 1=full convergence between individual’s understanding and ground truth) Consistency Degree of ‘deviation’ from understanding gained from previous time period Currency Time lag of understanding Precision Level of granularity of understanding Fitness for Use Measures quality in reference to criteria that are determined by the situation Measures Completeness Percentage of ground truth picture included in understanding Accuracy Degree to which precision matches what is needed (0=no match, 10=high degree of matching between precision level needed and available) Relevance Proportion of understanding that is related to task at hand Timeliness Degree to which currency matches what is needed (0=no match, 10=high degree of matching between currency level needed and available) Uncertainty Confidence level (0% =uncertain, 100%= certain) or confidence interval (95%, 90%, etc.) of awareness EBR RAND 8/3/2004 Slide 81
OFT The NCW Framework Initiative ASDC3I Key To Developing and Applying NCW Theory Across DoD Enterprise Board of Directors Code of Best Practice Tools NCW Enterprise Focused Methodology Applications Research NCW Theory (Hypotheses) NCW • Analysis Conceptual Knowledge Framework • Experiments Base • Exercises Metrics • Case Studies • T & E Relationships • Etc. NCW Awareness Focused Experiments Education Consulting EBR RAND 8/3/2004 Slide 83
OFT ASDC3I Status of Framework Development • “Getting NCW Theory and Metrics Right…” – Where we are • Second generation framework – Being evolved through peer review • Initial case study in progress (Air-to-Air combat) – Where we are going • Additional peer review • Continual revision • “…And Applied Enterprise-Wide” – Where we are • Establishing collaborative partnerships – Where we are going • Dissemination and education – Symposium, workshop, web, brochure, tutorials • Additional case studies – Sponsored, supported, encouraged EBR RAND 8/3/2004 Slide 84
OFT Application of the NCW Framework ASDC3I to Air-to-Air Combat • Objectives – Gain insight into how NCW is a new source of power in Air-to-Air combat – Illuminate contribution of enabling capabilities in the NCW value chain – Identify areas where data is needed – Assess utility of framework and identify needed improvements • Approach – Start with data from the JTIDS Operational Special Project – Apply NCW Framework to instantiate influence model – Capitalize on additional data and impute missing data – Identify sources of improved combat power EBR RAND 8/3/2004 Slide 85
Numerical Metrics for Case Study OFT ASDC3I Calculated with Analytica EBR RAND 8/3/2004 Slide 86
NCW Conceptual Framework OFT ASDC3I Information Value Added Force C2 Effectors Sources Services Degree of Networking Quality of Organic Information Degree of Information “Share-ability” Quality of Individual Information Degree of Shared Information Quality Degree of Shared Sensemaking Quality of Individual Sensemaking of Awareness Shared Awareness Inter- Understanding Shared Understanding actions Quality of Individual Decisions Quality of Collaborative Decisions Degree of Decision/ Plan Synchronization Degree of Actions/ Entities Synchronized Operating Environments Degree of Effectiveness/ Agility EBR RAND 8/3/2004 Slide 87
Quality of Organic Information: OFT ASDC3I Blue Tracks Blue11, 12 1 0.75 0.5 0.25 0 B B B B Q Q Q Q 1 2 3 4 1 0.75 1 0.5 0.75 0.25 0.5 0 B B B B Blue13, 14 Q Q Q Q 0.25 1 2 3 4 0 B B B B Q Q Q Q 1 2 3 4 B : Correctness: Location B : Completeness: Detection Q 3 Q 1 B : Correctness: ID Q 2 Q 4 B : Correctness: Velocity EBR RAND 8/3/2004 Slide 88
Influence of Networking on OFT ASDC3I Information “Share-ability” Force Information. Sources Value Added Services C2 Effectors Degree of Networking Quality of Organic Information Net Ready Nodes Network Fitness for Use Objective Measures Capacity Completeness Reach Correctness Connectivity Accuracy Quality of Service Consistency Currency P&R Capability Support Relevance Network Assurance Precision Collaboration Support Timeliness Network Agility Node Assurance Quality of Individual Information Fitness for Use Degree of Information Objective Measures Completeness “Share-ability” Correctness Quantity of Posted Info Accuracy Quality Consistency of Quantity of Retrievable Info Interactions Relevance Currency Ease of Use Timeliness Precision EBR RAND 8/3/2004 Slide 89
OFT ASDC3I Computing Quantity of Posted Info Force Information. Sources Value Added Services C2 Effectors Degree of Networking Whether nodes can transmit to Net Ready Nodes network Network Capacity Reach Posting channel numbers, types, and bandwidth (for data links Connectivity only) Quality of Service P&R Capability Support Prob of correct transmission Network Assurance Collaboration Support Adjustments to probability that Network Agility Exogenous variables: QoS will be delivered •Number of nodes Node Assurance •Number and size of files •Variables that impact how quickly nodes can transmit Degree of Information (CONOPS, coding schemes, “Share-ability” etc.) •Policies determining f(…) Quantity of Posted Info priority for posting •Expiration age for each type of info objects Quantity of Retrievable Info In this scenario, quantity of posted info. •Maximum queue lengths equals quantity of retrievable info, except for probability of hearing voice Ease of Use Voice Only Link 16 EBR RAND 8/3/2004 Slide 90
OFT Quantity of Retrieved Information ASDC3I Percent of Organic Information Retrieved Voice Only Link 16 + Voice Threat Tracks Blue Tracks Threat Tracks Blue Tracks 1 1 1 1 AWACS 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0.5 0.25 0.25 0.25 0.25 0 0 0 0 B B B B B B B B T T T T T T T T Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 1 1 1 Blue 11 & 12 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0.5 0.25 0.25 0.25 0.25 0 0 0 0 B B B B B B B B T T T T T T T T Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 1 1 1 Blue 13 & 14 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0.5 0.25 0.25 0.25 0.25 0 0 0 0 B B B B B B B B T T T T T T T T Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 Q Q Correctness: Location Completeness: Detection 3 1 Q Q Correctness: ID 4 Correctness: Velocity 2 EBR RAND 8/3/2004 Slide 91
OFT Comparing MCPs Using Summary Metrics ASDC3I Degree of Networking and Quantity of Information Retrieved Quality of Individual Information Degree of Degree of Shared Information Information “Share-ability” 1.0 0.08 Quality of 0.5 1.0 Degree of Individual Networking Sensemaking 0.28 Quality of Individual Quality of Decisions Organic Info Kill Ratio (Synchronization / Effectiveness) Voice Overall average over information quality * * Voice + Link 16 dimensions and package members EBR RAND 8/3/2004 Slide 92
Influence of Organic Info Quality and OFT ASDC3I Degree of Info Sharing on Quality of Individual Information Degree of Networking Quality of Organic Information Net Ready Nodes Network Fitness for Use Objective Measures Capacity Completeness Reach Correctness Connectivity Accuracy Quality of Service Consistency Currency P&R Capability Support Relevance Network Assurance Precision Collaboration Support Timeliness Network Agility Node Assurance Quality of Individual Information Fitness for Use Degree of Information Objective Measures “Share-ability” Completeness Correctness Quantity of Posted Info Accuracy Quality Consistency of Quantity of Retrievable Info Interactions Relevance Currency Ease of Use Timeliness Precision EBR RAND 8/3/2004 Slide 93
OFT Quality of Individual Information: ASDC3I Correctness Quality of Organic Information •Correctness of organic information Fitness for Use Objective Measures Completeness Correctness Exogenous variables: •Fusion Accuracy Consistency •Training •Whether the info was retrieved in original form Currency Relevance Precision •Whether the presentation Timeliness of the info introduces errors f(…) Quality of Individual Information Fitness for Use Degree of Information Objective Measures Completeness “Share-ability” Correctness Quantity of Posted Info Accuracy Consistency Quantity of Retrievable Info Relevance Currency Ease of Use Timeliness Precision Voice Only Link 16 EBR RAND 8/3/2004 Slide 94
OFT Quality of Individual Information: ASDC3I Voice Only vs. Link 16 Voice Only Link 16 + Voice Threat Tracks Blue Tracks Threat Tracks Blue Tracks 1 1 1 AWACS 1 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0.5 0.25 0.25 0.25 0.25 0 0 0 0 B B B B B B B B T T T T T T T T Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 1 1 1 Blue 11 & 12 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0.5 0.25 0.25 0.25 0.25 0 0 0 0 B B B B B B B B T T T T T T T T Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 1 1 1 Blue 13 & 14 0.75 0.75 0.75 0.75 0.5 0.5 0.5 0.5 0.25 0.25 0.25 0.25 0 0 0 0 B B B B B B B B T T T T T T T T Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 Q Q Correctness: Location Completeness: Detection 3 1 Q Q Correctness: ID 4 Correctness: Velocity 2 EBR RAND 8/3/2004 Slide 95
Comparing MCPs Using Summary Metrics OFT ASDC3I Quality of Individual Information Quality of Individual Information Degree of Degree of Shared 0.91 Information Information “Share-ability” 1.0 0.4 0.08 Quality of 0.5 1.0 Degree of Individual Networking Sensemaking 0.28 Quality of Quality of Individual Organic Info Decisions Kill Ratio (Synchronization / Effectiveness) Voice Overall average over information quality * * Voice + Link 16 dimensions and package members EBR RAND 8/3/2004 Slide 96
OFT ASDC3I Influence of Information “Share-ability” and Nature and Quality of Interactions on Degree of Shared Information Degree of Information “Share-ability” Quantity of Retrievable Info Ease of Use Quantity of Posted Info Quality of Interactions Depth Degree of Shared Information Quality of Individual Information Quantity Quality Fitness for Use Objective Measures Objective Measures Breadth Completeness Extent Correctness Reach Selectivity Intensity Quality Accuracy Continuity Latency Consistency Completeness Synchroniety Mode Correctness Agility Relevance Currency Accuracy Flexible Robustness Consistency Response Timeliness Precision Relevance Innovative Adaptability Currency Individual Characteristics Timeliness Quality of Individual Sensemaking: Awareness Competence Precision Risk Prop Trust Fitness for Use Degree of Shared Sensemaking: Shared Awareness Confidence Org. Ident Objective Measures Objective Measures Completeness Organizational Characteristics Correctness Extent Competence Risk Prop Accuracy Quality Confidence Trust Consistency Correctness Hardness Completeness Size Relevance Permanence Diversity Currency Consistency Accuracy Autonomy Structure Timeliness Interdependence Organizational & Ind. Behavior Currency Timeliness Precision Efficiency Cooperation Uncertainty Precision Uncertainty Synchronization EBR RAND 8/3/2004 Slide 97 T vs. T Deg. Of Eng
OFT ASDC3I Computing Extent of Shared Information Whether Whether information can Degree of Information “Share-ability” Quality of Interactions sender can be physically Depth Quantity of Posted Info communicate shared across Quantity Quality Quantity of Retrievable Info Breadth info with network Reach Selectivity Ease of Use receiver Intensity Continuity Latency Whether sender Degree of Shared Information Synchronicity Mode Objective Measures and receiver are Agility Extent: Proportion of force entities part of the same Robustness Flexible that share information f(…) collaborative Response Quality group Innovative Adaptability Completeness Correctness Individual Characteristics Risk Prop Competence Accuracy Trust Consistency Org. Ident Confidence Relevance Currency Organizational Characteristics Number of communications Timeliness Risk Prop Competence Precision “hops” between sender and Trust Confidence receiver Degree of Shared Sensemaking: Shared Awareness Size Hardness Objective Measures Diversity Permanence Extent Autonomy Structure Quality Interdepend Probability that sender Correctness Completeness Organizational &Individual will attempt to share Behavior information with Consistency Accuracy Cooperation Efficiency receivers Syn Currency Timeliness Deg. Of Eng T vs. T Precision Uncertainty EBR RAND 8/3/2004 Slide 98
OFT Degree of Shared Information: ASDC3I Extent of Shared Track Information Voice Only Voice + Link 16 Threat Tracks Threat Tracks All blue AC have 1 1 the same shared 0.75 0.75 information 0.5 0.5 in this example 0.25 0.25 (all listen to the 0 0 T T T T T T T T same voice Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 channel or receive the same Link 16 Blue Tracks Blue Tracks broadcasts) 1 1 0.75 0.75 0.5 0.5 0.25 0.25 0 0 T T T T T T T T Q Q Q Q Q Q Q Q 1 2 3 4 1 2 3 4 Q Q Correctness: Location Completeness: Detection 3 1 Q Q Correctness: ID 4 Correctness: Velocity 2 EBR RAND 8/3/2004 Slide 99
Comparing MCPs Using Summary Metrics OFT ASDC3I Degree of Shared Information Quality of Individual Information Degree of Degree of Shared 0.91 Information Information “Share-ability” 1.0 1.0 0.4 Quality of 0.08 0.22 0.5 1.0 Degree of Individual Networking Sensemaking 0.28 Quality of Individual Quality of Decisions Organic Info Kill Ratio (Synchronization / Effectiveness) Voice Overall average over information quality * * Voice + Link 16 dimensions and package members EBR RAND 8/3/2004 Slide 100
Influence of Information “Share-ability” and OFT ASDC3I Nature and Quality of Interactions on Individual Awareness, Understanding, and Decisions: Timeliness Degree of Information “Share-ability” Q of Retrievable Info Ease of Use Q of Posted Info Quality of Individual Sensemaking: Awareness Fitness for Use Degree of Shared Sensemaking: Shared Awareness Quality of Interactions Objective Measures Objective Measures Completeness Extent Depth Correctness Accuracy Quality Quantity Quality Completeness Consistency Correctness Breadth Relevance Consistency Distance Selectivity Accuracy Currency Timeliness Intensity Currency Timeliness Precision Uncertainty Continuity Latency Uncertainty Precision Synchronicity Mode Agility Quality of Individual Sensemaking: Understanding Degree of Shared Sensemaking: Shared Understanding Objective Measures Fitness for Use Robustness Flexible Objective Measures Extent Response Completeness Correctness Innovative Adaptability Quality Accuracy Correctness Completeness Consistency Individual Characteristics Relevance Consistency Accuracy Risk Prop Competence Currency Timeliness Currency Trust Timeliness Precision Org. Ident Confidence Uncertainty Precision Uncertainty Organizational Characteristics Quality of Individual Decisions Quality of Collaborative Decisions Risk Prop Competence Fitness for Use Fitness for Use Appropriateness Objective Measures Trust Confidence Agility Agility Objective Measures Appropriateness Completeness Extent Size Hardness Consistency Robustness Robustness Completeness Accuracy Diversity Permanence Consistency Currency Flexibility Flexibility Autonomy Structure Accuracy Currency Relevance Precision Responsiveness Responsiveness Interdepend Relevance Precision Timeliness Organizational & Ind. Behavior Innovativeness Innovativeness Timeliness Uncertainty Cooperation Efficiency Adaptability Adaptability Mode of D. M. Syn Uncertainty Mode of D. M. Risk Propensity Deg. Of Eng T vs. T EBR RAND 8/3/2004 Slide 101
Recommend
More recommend
