Master’s Project Final Presentation for IF-BAD ( I nstrument F or B udget A nalysis and D ecision-making) Fall 2018 Scott Kuhn Sasha Snisarevska Gang Xiang SYST 699
Agenda 1. Problem Definition 2. Scope 3. Requirements 4. Approach 5. Verification and Tool Use Examples 2
Agenda 1. Problem Definition 1.1 Sponsorship 1.2 Background 1.3 Problem Statement 1.4 Project Objectives 1.5 Stakeholders 2. Scope 3. Requirements 4. Approach 5. Verification and Tool Use Examples 3
Sponsorship ● SPA (Systems Planning and Analysis, Inc.) ○ Helps decision makers identify problems, assess risks, optimize resources, develop strategies, and test and refine ideas ○ Support a large variety of DoD and other government agencies missions ○ Created simulation and decision analytics tools and models ● SPA’s customer: SIAC (Strategic Intelligence Analysis Cell) ○ Analytic foundation for Department of Defense investment decisions to restore technical overmatch of the U.S. Armed Forces ● SIAC’s goals ○ Understand the enemy ○ Analyze U.S. capabilities ○ Track technology trends ○ Assess emerging threats 4
https://gallery.yopriceville.com/Free-Clipart-Pictures/Money-PNG/Money_Bag_PNG_Clipart_Picture#.W52j9uhKhPY Background ● Research, Development, Test and Evaluation (RDT&E) ○ Seven levels of budget activity ■ 01 – Basic Research ■ 02 – Applied Research ■ 03 – Advanced Technology Development ■ 04 – Advanced Component Development and Prototypes ■ 05 – System Development and Demonstration ■ 06 – RDT&E Management Support ■ 07 – Operational System Development ○ The data is publicly available for previous years and five years in advance ○ For example, the 2019 President’s Budget was released in February 2018 and contains figures for 2017 and 2018, as well as projected amounts for 2019-2023 ○ Federal Budget FY 2019 for defense is around $700 billion, and has increased for the last 4 years 5
Background cnt’d Two sources of data: ● Budget Database ○ President’s Budget data since 2011 ● President’s Budget ○ publicly available XML data files Budget Database 6
Background cnt’d ● Acquisition Process Milestones (Hagan) ○ Milestone A - Start tech/risk research and analysis ○ Milestone B - Start engineering/manufacturing work ○ Milestone C - Start production, deployment and operations support 7
Data Structure Service: Army Budget Activity: 5 Service Budget Activity Light Tactical Program Element PE#: 1 PE#: 3 Wheeled Vehicles Project P#: 1 LTV Prototype P#: 3 Suspension Accomplishment A#: 1 A#: 3 Improvement MS: B Milestone 8
Problem Statement Agencies conducting DoD research have milestones set years in advance for various capabilities they are developing, however their timelines may not be accurate, because they tend to slip or change over time. Publicly available budget sources hold timeline and milestone information, but there exists no tool that can analyze milestone data. 9
With what we started... Thousands pages of pdf/xml documentation... 10
Project Objectives 1. Develop a system that tracks predicted milestone dates across accomplishments, projects, program elements, and services. 2. Present milestone data across multiple years in an easy-to-read graphical element. 11
Agenda 1. Problem Definition 2. Scope 3. Requirements 4. Approach 5. Verification and Tool Use Examples 12
Scope ● RDT&E Budget Activity Numbers ○ 4 through 7 (Advanced Concept Development & Prototypes through Operational System Development) ● Procurement funding is outside the scope (done after RDT&E phase) ● 10/2/18 - Predicting accomplishment/milestone completion is outside scope 13
Stakeholders ● Department of Defense ○ Army, Navy, Air Force, DARPA ● DoD Laboratories ● Federally Funded Research & Development Centers and University-Affiliated Research Centers ● Industry 14
15
Agenda 1. Problem Definition 2. Scope 3. Requirements 3.1 Mission Requirements 3.2 System Requirements 3.3 Functional Requirements 4. Approach 5. Verification and Tool Use Examples 16
Mission Requirements MR1 : System shall visualize RDT&E milestone data, over a predetermined period System Requirements SR1 : System shall interface with existing milestone database sources SR2 : System shall plot milestones for specified accomplishments over time 17
Functional Requirements FR1 : System shall support a capability of milestone analysis based on historical data. FR2 : System shall provide a user-friendly Graphical User Interface (GUI). FR3 : System shall support searches and filters for accomplishments by keywords, project descriptions, program element, project, and program element number. FR4 : System shall display accomplishment descriptions FR5 : System shall display project descriptions 18
Agenda 1. Problem Definition 2. Scope 3. Requirements 4. Approach 4.1 Technical Approach 5. Verification and Tool Use Examples 19
Technical Approach 20
IF-BAD’s Capabilities ● Database connection ● User’s Settings ● Filtering ● Search ● Sorting ● Selection/Deselection ● Plotting Milestones 21
Filter Capability User must follow a hierarchical order when input the data such as: 1. Service Title 2. Budget Activity Number 3. Program Element Title 4. Project Title Pros: High accuracy Cons: User must know details about the searched item. 22
Search Engine The text search is carried out by Simple Search and Advanced Search . Simple Search: Single or multiple search for items of one type (e.g., titles, project numbers) Advanced Search: “AND” or “combined” search through items of different types ● Ignores Uppercase/Lowercase, spaces 23
24
Demo ● Database connection ● Filtering ● Sorting ● Selection/Deselection https://drive.google.com/open?id=1xzOx0ceqsBTemPWyvfKPvbrkP6kIOwfO 25
Get Milestones (1) 26
Levenshtein Distance ● Measure of similarity between two strings ● Number of deletions/insertions/substitutions required to transform one into the other ○ “GUMBO” and “GUMBALL” have a distance of 3 ○ Percent = 1 - Distance/LargestStringLength Ex: Why Levenshtein? User wants “JWARN 2” milestones: With String Matching Same accomplishment, 2015 2018 2014 different titles! 3) JWARN 2 JWARN Incr. 2 JWARN 2 With Levenshtein 27 (Gilleland, 2006)
Get Milestones (2) ● Inputs ○ Accomp. ID ○ Accomp. Title ○ Accomp.’s project’s Project ID ● Outputs ○ Formatted Output Data 28
Get Milestones(3) - Parse Milestones Out of Events ● Ignores special characters ● Avoids common non-milestone titles ● Looks through subproject, accomplishment, event titles 29 ● Handles acronyms
Assumptions ● All milestone events have “ms” or “milestone” inside the title ● Accomplishment acronyms/titles can change over time, but not dramatically ● Accomplishments have milestones only within their project/program element ● Users will select accomplishments that are meaningful ○ In use: when milestone title is generic, it belongs to the selected accomplishment if the accomplishment is in the list of accomplishments 30
Limitations & Solutions Limitations Solutions Levenshtein Criteria Sensitivity. Configurable in user settings, no Too high, can’t grab any milestones. need to touch code source Too low, grabs unrelated milestones. No strict format for PB publications. President’s User manual details how to edit Budget content changes over time. code source. System has many features (Search, Filters, User manual. Help tooltips. Code Source, Analysis). Is it usable? Considers similar accomplishments as one (Ex: JWARN 2 vs JWARN Incr. 2) 31
Agenda 1. Problem Definition 2. Scope 3. Requirements 4. Approach 5. Verification and Tool Use Examples 32
Verification Test 1/Test 2 Test 1 (Search, Filter, Select, De-select) 1. System is open on the Tool/Main Page 2. Search/Filter for this accomplishment by (Filter/Simple Search/Adv Search) 3. Select accomplishment 4. Double click to read the accomplishment’s description 5. Only once, deselect one accomplish. Re-add it 6. Repeat 2-4 for all test accomplishments Test 2 (Analysis, Hide, Unhide) 1. Do Test 1 2. Click “Start Analysis” 13 chosen test 3. Hide all accomplishments, then unhide them accomplishments of various formats (After Slides) 4. Check President’s Budget PDF files to confirm milestone analysis accuracy 33
Verification System has met 6/6 requirements ● SR1 - Connect with DB - Test 1 ● SR2 - Plot Milestones - Test 2 ● F1 - Milestone Analysis - Test 2 ● F2 - GUI - Inspection/Test 2 ● F3 - Searches/Filters - Test 1 ● F4 - Accomplishment Descriptions - Test 1 ● F5 - Project Descriptions - Test 1 34
Verification - Derived & Extra Met 22/24 Derived or Extra Requirements Missing Extra Requirements: ● Text Analysis of Descriptions to Find Source of Slippage ● Allow Users to Manually Link Accomplishments 35
Recommend
More recommend
