Building a Science of Open Source Analysis: Addressing the need for real tradecraft using open source analysis
Current Context of Intelligence Analysis • Customers and consumers demanding greater understanding of political and national security contexts • Demand for more and BETTER intelligence • Customers: Greater fidelity on issues of the day • Consumers: Just In Time products and services to help shape policy and formulate action • More visibility into quality of products and services they receive, including precisely how analysts come to their conclusions • Challenge for LAS: Identify ways to make analysts better by helping them create more and better intelligence, in compliance with ICD 203 and the requirements of ODNI Analytic Integrity and Standards group
Before we begin – Words and how we will use them… Term Our Operational Definition Science Science is more than the mechanics of careful observation, creating hypotheses, testing those hypotheses against data, replicating and validating those findings and/or reformulating hypotheses in the pursuit of understanding some phenomena. Science is also the mechanism of organizing a corpus of knowledge and turning a community of interest into a community of practice. Science is the the common language and formal communication that results in knowledge sharing within those communities of practice. Science is also an agreed standard against which practitioners and their findings are judged and incorporated into the corpus of knowledge. SOSA Science of Open Source Analysis
A few more words… Term Our Operational Definition OSINT “Publicly available information appearing in print or electronic form including radio, television, newspapers, journals, the Internet, commercial databases, and videos, graphics, and drawings. While open-source collection responsibilities are broadly distributed through the IC, the major collectors are the DNI's Open Source Center (OSC) and the National Air and Space Intelligence Center (NASIC).”* Open Source Analysis Using open source data (including OSINT) to inform all source and single source analysis True Factually Accurate Confidence in Source Degree to which source is legitimate, reliable Confidence in Data Degree to which data are true, rigorously determined Confidence in Judgment Degree to which analytic findings are likely, based on examination of evidence trail; logical flow of thought processes *As defined by ODNI at http://www.dni.gov/index.php/about/faq?start=2, retrieved 2/15/16 4
... and Intelligence Collection Disciplines* Discipline Acronym Type of Data Collected Responsible Agency Signals SIGINT Derived from signal intercepts comprising - individually or combined: National Security Agency (NSA) Intelligence communications intelligence (COMINT), electronic intelligence (ELINT) and foreign instrumentation signals intelligence (FISINT). Imagery IMINT Representations of objects reproduced electronically or by optical National Geospatial-Intelligence Intelligence means on film, electronic display devices, or other media. Can be Agency (NGA) derived from visual photography, radar sensors, and electro-optics. Measure- MASINT Technically derived data other than imagery and SIGINT. Locates, Defense Intelligence Agency (DIA), ment and identifies, or describes distinctive characteristics of targets. It Directorate for MASINT and Signature employs disciplines including nuclear, optical, radio frequency, Technical Collection (DT) intelligence acoustics, seismic, and materials sciences. Human HUMINT Derived from human sources. Collection includes clandestine Central Intelligence Agency (CIA), Intelligence acquisition of photography, documents, and other material; overt Directorate of Operations (DO) collection by personnel in diplomatic and consular posts; debriefing of foreign nationals and US citizens who travel abroad; and official contacts with foreign governments. Open OSINT Publicly available information -print or electronic -including radio, DNI Open Source Enterprise (OSE) Source television, newspapers, journals, the Internet, commercial databases, National Air and Space Intelligence and videos, graphics, and drawings. Intelligence Center (NASIC). Geospatial GEOINT Analysis and visual representation of security-related activities on the National Geospatial-Intelligence Intelligence earth. Integration of imagery, IMINT, and geospatial information. Agency (NGA) *As defined by ODNI (http://www.dni.gov/index.php/about/faq?start=2), retrieved 2/15/16 5
Overview of Major Elements of Intelligence Analysis Collected Sources HUMINT SIGINT NIPF/PIP Intelligence MASINT IMINT Political Political Questions Stability Stability GEOINT OSINT Provide Provide International International Situational Situational Ambition Ambition Intelligence Analysis awareness awareness Product Economic Economic Stability Stability Warn of Warn of Alternative Think Tanks Media situation change situation change Societal Societal Issues Issues Explain Explain specific event specific event Academia NGOs OSA Crime and Crime and Trafficking Trafficking New/ Business Emerging Intelligence Media OSA Methods Can Vary Based on National Priorities and/or Intelligence Questions Collectible Sources 6
A View of the Intelligence Analysis Production Process Understanding analytic production processes is necessary but insufficient for understanding what happens ‘between the ears’ of the analyst. That is STILL not well understood. 7
Taking an illustrative look in the ”black box”
A good analyst is an accomplished learner. Evaluation Conceptual Framework: Bloom’s Taxonomy of Learning Objectives (1956) Synthesis Analysis Application Comprehension Knowledge
Understanding Self – The Analyst’s Lens • Knowledge, Skills, Ability • Education • Experience • Type of Analyst • Agency • On the job learning • Professional Network • National Security Worldview – purpose of intelligence and its relation to policymakers
Intelligence Question Is Iran violating the nuclear agreement? Question Decomposition 1.What do I need to know to What constitutes What is in the answer this question? Iran violating? agreement? Nuclear 2.What do I need to know Religion Deception Agreement Manufacturing about these topics to answer the question? Finance Using loopholes Basic International Science & Policy Culture Politics History Technology Non-Compliance Monitoring Military Business Leadership Weapons Enforcement Economics
Resource Identification Judgment Personal Assessment 4. What information do I need? HUMINT SIGINT 5. What information do I have? 3. What knowledge/skills do I bring 6. What information don’t I Subject to this task vice the Matter MASINT have? IMINT knowledge/skills that are needed? Experts? 7. Are the data factually GEOINT accurate? OSINT 8. How confident am I that the data sources are reliable? Alternative Think Tanks Media Academia NGOs New/ Business Emerging Intelligence Media
Conundrum Evaluation Synthesis 9. What do I do with the data? 10. What are the results? 12. How do I tell How do I decide - What do they mean? the story with the findings I have? What methods? 11. How do I know the What tools? 13. How do I tell the results are reliable? story to address the What techniques? audience who asked the question? There’s a lot of work for us to do.
Our Research Agenda at LAS Challenge: Identify ways to make analysts better by helping them create more and better intelligence Investigate what goes on ‘between the ears’ of analysts • Analyst Assessment Support Program – Using ”Expert in a Box” approach • Building a Science of Open Source Analysis 14
Analyst Assessment Support Program • Previous research • Developed evidence-based approaches for social science problem sets • Problem recognition and problem solving in complex environments • Created method that functions as decision-aide for analysts by providing expert information based on evidence from empirical research • Includes guided approach to analysis, source citation collection, reporting function • Trained, tested method for reliability (inter-rater) and validity (content) • Successfully implemented for use with a number of different environments, requiring only minimal adjustments • Developed decision aids for health science problem sets • Conducted numerous task and cognitive task analyses • LAS research • Bring expert aspect (research findings) in method up to date, redesign tool • Identify how this method can be improved by or act as a complement to LAS projects, such as Open KE, FSP, others • Test the updated product 15
Using established academic disciplines to inform SOSA • Previous research • Ethnography of intelligence analysts • Visualizing PMESII • Socio-Cultural Influences • Multi-layered/multi-disciplinary analytic methods • Structured models, approaches, and techniques • LAS research • Supply Chain Analysis and Intelligence Analysis (DO5) • Conduct a Use Case to identify the interaction of supply chain analysis and other methods to analyze adversarial supply chains re: nuclear proliferation 16
Recommend
More recommend
