TAC KBP 2016 Linguistic Resources: Event Arguments (EA), Event Nuggets (EN) and Belief/Sentiment (BeSt) Joe Ellis (presenter), Jennifer Tracey (presenter), Jeremy Getman, Zhiyi Song, Ann Bies, Stephanie Strassel Linguistic Data Consortium University of Pennsylvania
Introduction and Overview Linguistic resources for TAC KBP 2016 Eighth year LDC produced KBP resources Twenty-nine new data sets Two primary goals Increase coordination across tracks Increase multi-lingual evaluation tracks Yesterday Doc selection, ED&L, and Cold Start Today Event Arguments, Event Nuggets, and Belief/Sentiment (BeSt) TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
Entities, Relations, & Events (ERE) Entities, Relations, and Events (ERE) Ongoing annotation task developed by LDC for DARPA’s Deep Exploration and Filtering of Text program (DEFT) Exhaustive labeling of entities, relations and events and their attributes. ERE annotation performed as upstream task Provided inputs for multiple downstream tasks supporting ED&L, EA, EN, and BeSt Primary means of meeting increased coordination of data goal TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
ERE Annotation Entity Event Type.Subtype Realis Trigger Arguments The Bo Xilai event was ignited NAM Filler (Rich ERE) Bo Xilai, Bo Xilai H1 Movement. Actual running Lijun PER PRO his Relation SPC IND Time Transport 2012-XX-XX US by Lijun running into the US Person PER NAM Crime bribery and corruption Lijun SPC IND R1 Physical.Located Lijun consulate NAM Bogu Kailai consulate in 2012 to bring Bogu US consulate into 2012 PER NOM his wife SPC IND his R2 Personal.Family H2 Conflict. Actual killing Bogu Kailai Kailai’s killing to light. Will Bo SPC IND GPE NAM US wife Attack his wife Xilai end up in jail due to SPC IND LOC NOM H3 Justice. US consulate Other jail Bo Xilai Jail Non bribery and bribery and corruption; what will corruption LOC NOM jail SPC his wife end up with? H4 Life.Die Actual killing Bogu Kailai H5 Movement. Other end up Bo Xilai Transport jail Person TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
ERE Event Types ERE event type inventory reduced for 2016 8 types and 18 subtypes in 2016 (listed below) 9 types and 38 subtypes in 2015 (listed in overview paper) Most of the dropped event types and subtypes are scarce in existing data Conflict.Attack Manufacture.Artifact Justice.ArrestJail Conflict.Demonstrate Movement.TransportArtifact Life.Die Contact.Broadcast Movement.TransportPerson Life.Injure Contact.Contact Personnel.Elect Transaction.Transaction Contact.Correspondence Personnel.EndPosition Transaction.TransferMoney Contact.Meet Personnel.StartPosition Transaction.TransferOwnership TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
ERE annotation counts 4000 3500 3000 2500 2000 1500 1000 500 0 NW DF NW DF NW DF Chinese Chinese English English Spanish Spanish Entities Fillers Relations Event Mentions Event Hoppers TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
ERE event mentions 700 600 500 400 300 200 100 0 CMN ENG SPA TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
Event Argument (EA) Overview Given new approach to evaluating EA in 2016, data development procedure overhauled 2014-2015 Manual run Argument-level assessment 2016 Gold Standard Event-level cross-document task • Queries, manual run, assessment TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
EA Gold Standard ERE annotations on core source corpus Augmentation pass BBN script run over ERE data Annotators review results Inferred arguments Locational containment Not annotated in ERE Baghdad as Place of Conflict.Attack Iraq added as 2 nd Place TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
EA Cross-Document Queries & Manual Run Query selection 51 simple, low-granularity queries Event arguments in EAL Gold Standard Annotators reviewed over 1300 potential queries Manual Run Exhaustive across full 30K English source corpus Justification strings indicating presence of event hopper in doc Personnel.End-Position Person – Thabo Mbeki TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
EA Cross-document Assessment and Results Does justification prove presence of query in document? Correct: Response contains query event Event Type Match: Contains event of same type as query, but not query event Wrong: Doesn’t contain query event or event of same type Low system recall on manually selected queries BBN produced 249 “derived” queries based on system responses No LDC manual run for these queries TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
EA Cross-Document Results 7000 700 6000 600 5000 500 4000 400 300 3000 2000 200 1000 100 0 0 Systems LDC Systems CORRECT ET_MATCH WRONG CORRECT ET_MATCH WRONG Manual Queries Derived Queries TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
Event Nuggets and Linking (ENL) No separate ENL annotation task Data are entirely produced by running a script over ERE data to extract and reformat a subset for use by ENL 3000 2500 2000 1500 1000 500 0 NW DF NW DF NW DF CMN CMN ENG ENG SPA SPA Event Nuggets Event Hoppers TAC KBP 2016 Evaluation Workshop - NIST, November 14-15
Belief and Sentiment Annotation Only ERE entities are holders of belief and sentiment Only ERE entities, relations and events are targets of belief and sentiment For events only, belief marked for each argument as well as the event itself Belief values: committed, non-committed, reported, n/a Polarity also marked Sentiment values (polarity): positive, negative Sarcasm flag indicated when polarity annotated is opposite of literal meaning (based on context) TAC KBP Workshop, November 14-15, 2016
Belief Annotation Example Ominous new action by U KRAINE ’ S SECURITY FORCES on Monday, including a raid on AN OPPOSITION PARTY ’ S HEADQUARTERS , appeared to diminish prospects for talks between THE GOVERNMENT and Event: raid PROTEST LEADERS , as W ESTERN OFFICIALS grasped for CB a way to defuse THE COUNTRY ’s intensifying political Event: talks crisis. Arguments also NA Relation: Ukraine’s CB Arguments also security forces NA CB TAC KBP Workshop, November 14-15, 2016
BeSt Data Overview Language Belief annotations Sentiment annotations Training Evaluation Training Evaluation # Ann #/Doc # Ann #/Doc # Ann #/Doc # Ann #/Doc Chinese 13,192 66 12,163 76 27,982 140 18,982 118 English 18,915 77 21,188 128 38,664 157 25,358 154 Spanish 9,406 99 12,546 75 14,299 151 17,353 103 English evaluation data notably more dense in belief annotations than training data Spanish evaluation data less dense than training data in both belief and sentiment TAC KBP Workshop, November 14-15, 2016
Comparison of Training and Eval Data - Belief 16000 14000 12000 10000 8000 6000 4000 2000 0 Committed Non-Committed Reported N/A Committed Non-Committed Reported N/A Committed Non-Committed Reported N/A CMN CMN CMN CMN ENG ENG ENG ENG SPA SPA SPA SPA Training Evaluation TAC KBP Workshop, November 14-15, 2016
Comparison of Training and Eval Data - Sentiment 35000 30000 25000 20000 15000 10000 5000 0 Positive Negative None Positive Negative None Positive Negative None CMN CMN CMN ENG ENG ENG SPA SPA SPA Training Evaluation DEFT PI Meeting, May 28-29, 2015 – Boulder, CO
Recommend
More recommend
