Integrated Crisis Early Warning System (ICEWS) Computational Social - PowerPoint PPT Presentation

Integrated Crisis Early Warning System (ICEWS) Computational Social Science Experimental Proving Ground (CSS:EPG) Dr. Sean O’Brien December 29, 2008

Program Summary Objective – Create a comprehensive , integrated , automated , validated , analytic system that forecasts shifts toward/away from country/regional instability Approach – Monitor and assess, in near-real time, events and trends that may trigger crises – Develop and integrate validated, transparent, scientifically rigorous, robust and replicable models from diverse perspectives across multiple domains/levels of analysis February 2007 Solicitation for ICEWS September 2007 Contract Awards October 2007 Program start December 2008 Phase 1 Go/No-Go and Phase 2 Option Decisions March 2009 Phase 2 Options Awarded June 2010 Phase 3 Option Decisions September 2010 Phase 3 Options Awarded 1 Distribution authorized to U.S. Government Agencies only

Performers and Locations Arlington, VA Arlington, VA Atlanta, GA Arlington, VA Cambridge, MA Oahu, HI Lawrence, KS Philadelphia, PA Albany, NY Vienna, VA Weston, MA Syracuse, NY Washington, DC Philadelphia, PA Hilliard, OH Lexington, SC College Park, MD Aspen, CO Athens, GA Seattle, WA Wethersfield. CT Vienna, VA Washington, DC 2 Distribution authorized to U.S. Government Agencies only

LM-ATL Raven System Concept Stability Assessment & Mitigation Planning Primary Phase I Focus Interactive Interfaces to Extension of Phase I View & Query Forecasts and Mitigation Strategies Primary Phase II Focus Futures Exploration Events of Interest Overall Stability Aggregation, Explanation, News Blogs & Databases Feeds Reports & Transparency Key Indicators Govt. & Leaders Data Ingest Services Institutions Groups & Event Data Economics Factions Coding Collection (KU, SAE) Models and Model Services Text SME Forecasting Stability Model Processing Interview & Statistical Agent Based Bayesian Inputs & Analytics QC Tools Models Models Models (KU, SAE, UW, EG) (UP, LC, LM) (IDI, EG, SAE) DIME Action DIME Action DIME Action Exploration Region/Countries Regional Players Stability Impacts Mining Models DIME (DIME Effects) (DIME Strategies) (Linkages to Forecasting (LM) (IDI, LM, UP) Actions Models’ Levers) (LC, LM) (UP) DIME Action Modeling DIAS Framework (ANL) adapted by LM LM 3 Distribution authorized to U.S. Government Agencies only

Phase 1 Achievements Largest data set ever collected/analyzed for instability forecasting project – 6.5M news stories from 75 national, regional, and international sources (253M lines of text) • AP, UPI, BBC Monitor • India Today, Jakarta Post, Pakistan Newswire, Saigon Times – 100 other sources of data on country social, demographic, economic, leadership and political factors • The Economist Intelligence Unit, Freedom House, IMF, World Bank, Political Instability Task Force, and the Correlates of War project Developed fully automated capability to monitor and forecast political activity around the globe – Automatically convert news reports into structured indices that reflect the character and intensity of interactions between key leaders, organizations, and countries —who is doing what to whom, when, where and how around the world • Using actor dictionary (with over 8,000 entries) and a verb-phrase dictionary (with over 15,000 entries), news stories coded in four major categories—verbal cooperation/conflict, material cooperation/conflict—comprising 130 variables to measure and monitor the character and intensity of a broad range of political activities – Resulting indices are used in computer models to identify trends, and cue analysts to impending conflicts By aggregating some of the indices, performers were able to successfully forecast a variety of instability events in the Pacific Command area of responsibility – Though only one performer passed Phase 1 gates Some novel new insights 4 Distribution authorized to U.S. Government Agencies only

Phase 1 New Insights on Forces Driving Country Instability Identified and demonstrated a “repulsion” effect based on social similarity connectedness – In countries with similar temporal event patterns, the presence of social unrest in one country, reduces its likelihood in connected countries. Empirically confirmed inverted “U” relationship between government repression and probability of ethnic/religious violence – However, effects are conditional on the ethnic composition of the society 1 .004 ELF 0.5 .16 ELF .86 ELF 0 .6 ELF .36 ELF .16 ELF 1 .004 ELF 2 3 4 5 Political Terror Scale Weakness in the combined strength of the two dominant rival parties is a leading indicator for higher violence 5 Distribution authorized to U.S. Government Agencies only

Evaluation Methodology GFI Data: “Authoritative” set of data on the occurrences and intensity of de-stabilizing events in 29 countries of the PACOM AOR for the period 1998-2006 – Intensity levels 0-4* (common yardstick) • 0 – No crisis • 2 – Crisis, violence is used sporadically • 3 – Severe crisis, violence is used repeatedly and in an organized way • 4 – War, violence used with continuity, in an organized and systematic way – De-stabilizing events (variable; best 3 scored) • Domestic Political Crisis – Significant opposition to the government, but not to the level of rebellion or insurgency (e.g. power struggle between two political factions involving disruptive strikes or violent clashes between supporters) • Rebellion – Organized opposition whose objective is to seek autonomy or independence • Insurgency – Organized opposition whose objective is to overthrow the central government • Ethnic/Religious Violence – Violence between ethnic or religious groups that is not specifically directed against the government • International Crisis – Conflict between two or more states or elevated tensions between two or more states that could lead to conflict – Data for 1998-2004 provided to performers for training – Data for 2005-2006 withheld for testing *Index of Instability/Conflict Intensity from the Heidelberg Institute for International Conflict Research 6 Distribution authorized to U.S. Government Agencies only

Evaluation Methodology Performers calculate probability of 4 intensity levels for each country 6 mos hence – Bin probabilities using the 2/3 rule to determine “Low”, “Moderate” and “High” intensity – Compare performers matrix with true matrix for accuracy, recall and precision Max Max Max Max Max Sum of Probabilities >=2/3, Country Year Quarter Intensity = 0 Intensity = 1 Intensity = 2 Intensity = 3 Intensity = 4 Forecast = High Intensity 1 0.0192 0.4399 0.5343 0.0065 2005 0.0489 0.1817 2 0.5238 0.2456 Sum of Probabilities >=2/3, 3 0.1807 0.3831 0.3587 0.0774 Burma 4 0.3012 0.4243 0.2456 0.0288 Forecast = Moderate Intensity 1 0.1478 0.3432 0.4087 0.1003 2006 2 0.1700 0.3826 0.3683 0.0791 Sum of Probabilities >= 2/3, 0.2492 0.4085 0.2939 0.0484 3 Forecast = Low Intensity 4 0.3143 0.4450 0.2184 0.0224 Performers calculate probability of discrete de-stabilizing events for each country 6 mos hence – Use 2/3 rule to determine “0” or “1” – Compare performers matrix with true matrix for accuracy, recall and precision – Best forecasts for 3 de-stabilizing events will we scored and reported Probability <= 1/3, Forecast = No Domestic Ethnic/ International Rebellion Country Year Quarter Rebellion Insurgency Political Religious Crisis Crisis Violence 0.0004 1.0000 0.0000 0.4657 1 0.0027 Probability >=2/3, Forecast = 2005 2 0.0034 0.2269 0.8918 0.0000 0.4657 Insurgency Bangladesh 3 0.0000 0.0682 0.9999 0.0000 0.0438 4 0.0050 0.1720 0.9910 0.0000 0.0438 1 0.0034 0.9992 0.9996 0.0000 0.4657 1/3 < Probability < 2/3, No 2006 0.0013 0.4501 0.9979 0.0005 0.4116 2 forecast 0.0000 0.1695 0.9974 0.0000 0.8099 3 4 0.0001 0.9999 0.9943 0.0000 0.8099 7 Distribution authorized to U.S. Government Agencies only

Forecasting-Performance Metrics GO/ NO GO Forecast Window Recall Accuracy Precision Phase 80% Phase 0 80% 70% Annual (Benchmark) (Retrospective) > 80% Phase 1 80% 70% 1-6 mos. (Retrospective) >85% Phase 2 80% 70% 1-3 mos. (Real-Time) >85% Phase 3 80% 70% 1 mo. (Real-Time) 8 Distribution authorized to U.S. Government Agencies only

Results: LM-ATL Accuracy Recall Precision 100% Accuracy & Recall 80% Threshold 60% Precision Threshold 40% 20% 0% Max Domestic Ethnic/Religious International HI Reb Insur DPC ERV IC Rebellion Insurgency Intensity Political Crisis Violence Crisis Exceeds metrics for the maximum intensity index and 3 instability events: Rebellion, Insurgency, and Ethnic/Religious Violence – Passes Phase 1 gates By integrating improved versions of best of breed models from multiple perspectives, team achieves more accurate, precise forecasts than any one model alone 9 Distribution authorized to U.S. Government Agencies only