Social Event Extraction: Inferring International Relations and - PowerPoint PPT Presentation

Social Event Extraction: Inferring International Relations and Police Killings from the News Brendan O’Connor College of Information and Computer Sciences University of Massachusetts Amherst http://brenocon.com Data Science Workshop on Computational Social Science Yale University, October 20, 2017 Joint work with: Katherine Keith, Abram Handler, Michael Pinkham, Cara Magliozzi, Joshua McDuffie, Saul Shanabrook, Brandon Stewart, Noah Smith 1 Saturday, November 25, 17

Computational Social Science 1. Computationally mediated human social behavior: e.g. crowdsourcing, online auctions 2. Computationally oriented analysis methods: e.g. agent-based simulations 3. Artificial intelligence (ML/Vision/NLP) as a social scientific, data analysis method 2 Saturday, November 25, 17

Computational Social Science Official social data Data collection Data analysis 100 BCE 1829 1900 2000 3 Saturday, November 25, 17

Computational Social Science Official social data Newly available social data Data collection Data analysis Digitized behavior Billions of users Billions of messages/day Digitized news Thousands of articles/day 100 BCE 1829 Digitized archives Millions of books/century 1900 2000 3 Saturday, November 25, 17

TextGenerator ( SocialAttributes ) → Text Society (SocialAttributes) Writing (TextGenerator) Text Data (Text) Data generation process Language generation as social process P(TextGen | Text, SocAttr) Language for social measurement P(SocAttr | Text, TextGen) 4 Saturday, November 25, 17

TextGenerator ( SocialAttributes ) → Text Society (SocialAttributes) Writing (TextGenerator) Text Data (Text) Data generation process Social media usage Public opinion Model assumptions Language for social measurement P(SocAttr | Text, TextGen) [O’Connor et al., 2010] 5 Saturday, November 25, 17

TextGenerator ( SocialAttributes ) → Text Society (SocialAttributes) Writing (TextGenerator) Text Data (Text) Data generation process Real-world News media process political events Model assumptions Language for social measurement Israeli − Palestinian Diplomacy P(SocAttr | Text, TextGen) A : Israel-Jordan Peace C : U.S. Calls for West Bank Treaty Withdrawal 0.8 B : Hebron Protocol D : Deadlines for Wye River Peace Accord E : Negotiations in Mecca F : Annapolis Conference A B C D E F 0.4 0.0 [O’Connor, Stewart, Smith 2013] 1994 1997 2000 2002 2005 2007 6 Saturday, November 25, 17

TextGenerator ( SocialAttributes ) → Text Society (SocialAttributes) Writing (TextGenerator) Text Data (Text) Data generation process Real-world News media process political events Language generation as social process P(TextGen | Text, SocAttr) 7 Saturday, November 25, 17

TextGenerator ( SocialAttributes ) → Text Society (SocialAttributes) Writing (TextGenerator) Text Data (Text) Data generation process Geography Social media usage of authors weeks 1 − 50 weeks 51 − 100 weeks 101 − 150 − __ − Language generation as social process P(TextGen | Text, SocAttr) ctfu [Eisenstein et al. 2010, O’Connor et al. 2010, Eisenstein et al. 2012] 8 Saturday, November 25, 17

TextGenerator ( SocialAttributes ) → Text Society (SocialAttributes) Writing (TextGenerator) Text Data (Text) Data generation process Racial demographics Social media usage of authors Language generation as social process P(TextGen | Text, SocAttr) African-American English on Twitter Dialects and social media NLP [Blodgett et al. 2016, Blodgett and O’Connor 2017] 9 Saturday, November 25, 17

Police killings July 17, 2014 Aug 9, 2014 July 5, 2016 July 6, 2016 10 Saturday, November 25, 17

Police killings Data? New York, Eric Garner NY July 17, 2014 Michael Ferguson, Aug 9, 2014 Brown MO Alton Baton July 5, 2016 Sterling Rouge, LA July 6, 2016 Philando Falcon Castile Heights, MN 10 Saturday, November 25, 17

Police killings Data? New York, Eric Garner NY Michael Ferguson, Brown MO Alton Baton Sterling Rouge, LA Philando Falcon Castile Heights, MN 11 Saturday, November 25, 17

Police killings Data? • Are there more or fewer New York, Eric Garner NY fatalities than last year? Michael Ferguson, Brown MO Alton Baton Sterling Rouge, LA Philando Falcon Castile Heights, MN 11 Saturday, November 25, 17

Police killings Data? • Are there more or fewer New York, Eric Garner NY fatalities than last year? • Is there racial disparity/ discrimination? Michael Ferguson, Brown MO Alton Baton Sterling Rouge, LA Philando Falcon Castile Heights, MN 11 Saturday, November 25, 17

Police killings Data? • Are there more or fewer New York, Eric Garner NY fatalities than last year? • Is there racial disparity/ discrimination? Michael Ferguson, • Which police departments are Brown MO better or worse? What policing strategies are most effective or safe? Alton Baton Sterling Rouge, LA Philando Falcon Castile Heights, MN 11 Saturday, November 25, 17

Police killings Data? • Are there more or fewer New York, Eric Garner NY fatalities than last year? • Is there racial disparity/ discrimination? Michael Ferguson, • Which police departments are Brown MO better or worse? What policing strategies are most effective or safe? Alton Baton Sterling Rouge, LA • Need good data for the public interest and social science / policy making Philando Falcon Castile Heights, MN 11 Saturday, November 25, 17

Issues in government data • Washington Post, Oct. 16, 2016: “Americans actually have no idea” about how often police use force because nobody has collected enough data. 12 Saturday, November 25, 17

Offjce of Justice Programs Issues in government data • Unreliable partial compliance FIGURE 1 between local agencies and Estimated number of law enforcement homicides and percent not reported, by data source, 2003–2009 federal government and 2011 • Massively undercounts deaths Number of homicides reported Percent of expected deaths not reported to the ARD or SHR 8,000 [Banks et al. 2015 (BJS/DOJ), 7,000 7,427 51% 54% 28% (100%) 6,000 Lum and Ball 2015 (HRDAG, 5,000 5,324 external)] (72%) 4,000 3,620 3,385 3,000 (49%) (46%) 2,000 • [Compare: voluntary 1,000 0 participation approaches, e.g. Estimated Reported Reported Reported to universe to ARD to SHR either ARD or SHR National Justice Database] 13 Saturday, November 25, 17

Alternative: news media reports • Populate a database by manually reading news articles (filtered by keyword search) • FatalEncounters.org, KilledByPolice.net, The Guardian, Washington Post... • FE: volunteers have read 2M articles or ledes (!) • Augment with open records requests • BJS, Dec. 2016: media reports double the count compared to previous government collection efforts • Secondary vs primary sources 14 Saturday, November 25, 17

Computational approach • Goal: extract fatality records from a news corpus • Off-the-shelf event extractors work poorly (ACE, FrameNet training/ontologies) • Instead, train models for this problem (distant supervision+EM) • NLP and social analysis • Concrete, real-world tasks useful testbed for NLP research • Can NLP offer something useful for important tasks? • Public data and government accountability 15 Saturday, November 25, 17

Computational approach New York, Eric Garner NY July 17, 2014 Michael Ferguson, Aug 9, 2014 Brown MO Alton Baton July 5, 2016 Sterling Rouge, LA July 6, 2016 Philando Falcon Castile Heights, MN 16 Saturday, November 25, 17

Task: Database Population Time-delimited corpus Alton Baton Sterling Rouge, LA Infer names of persons killed by police during that Philando Falcon timeframe Castile Heights, MN 17 Saturday, November 25, 17

Task: Database Update New York, Eric Garner NY Michael Ferguson, Historical data Brown MO (Distant supervision) Alton Baton Sterling Rouge, LA Testing/Runtime Philando Falcon Castile Heights, MN 18 Saturday, November 25, 17

Mentions The Baton Rouge Police Department confirms that confirms Alton Sterling , 37 , died during a shooting at the Triple S Food Mart ... the two officers involved in Tuesday 's shooting of Alton Sterling ... ... Alton Sterling was a resident of Baton Rouge... 19 Saturday, November 25, 17

predict: describes Mentions police fatality? The Baton Rouge Police Department confirms that confirms Alton Sterling , 37 , died 0.4 during a shooting at the Triple S Food Mart ... the two officers involved in Tuesday 's 0.8 shooting of Alton Sterling ... ... Alton Sterling was a resident of Baton 0.01 Rouge... 19 Saturday, November 25, 17

predict: describes Mentions police fatality? The Baton Rouge Police Department confirms that confirms Alton Sterling , 37 , died 0.4 during a shooting at the Triple S Food Mart ... the two officers involved in Tuesday 's 0.8 shooting of Alton Sterling ... ... Alton Sterling was a resident of Baton 0.01 Rouge... aggregate: add to database? 19 Saturday, November 25, 17