Large Scale QA-SRL Parsing Nicholas FitzGerald, Julian Michael, - PowerPoint PPT Presentation

Large Scale QA-SRL Parsing Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer ACL 2018 http://qasrl.org/

Semantic Role Labelling Subject Manner Verb Object Time John surreptitiously ate the burrito at 2am.

Semantic Role Labelling Subject Manner Verb Object Time John surreptitiously ate the burrito at 2am. • Applied to improve state-of-the-art in NLP tasks such as Question Answering [Shen 2007] and Machine Translation [Liu and Gildea, 2010]

Semantic Role Labelling Subject Manner Verb Object Time John surreptitiously ate the burrito at 2am. • Applied to improve state-of-the-art in NLP tasks such as Question Answering [Shen 2007] and Machine Translation [Liu and Gildea, 2010] • Commonly used interface to facilitate Data Exploration and Information Extraction [ Stanovsky et al 2018] [Chiticariu et al. 2018]

Semantic Role Labelling Subject Manner Verb Object Time John surreptitiously ate the burrito at 2am. • Applied to improve state-of-the-art in NLP tasks such as Question Answering [Shen 2007] and Machine Translation [Liu and Gildea, 2010] • Commonly used interface to facilitate Data Exploration and Information Extraction [ Stanovsky et al 2018] [Chiticariu et al. 2018] • Considerable interest in general-purpose SRL parsers

QA-SRL Subject Manner Verb Object Time John surreptitiously ate the burrito at 2am. ? ? n n e e t ? t a a g e ? e n n g i g e h n n t t i a e i h h e m t t e e s o m m a s w o o e s s t t a s a s a h o a w W h w W n w e o h H W [He et al. 2015]

QA-SRL Subject Manner Verb Object Time John surreptitiously ate the burrito at 2am. ? ? n n e e t ? t a a g e ? e n n g i g e h n n t t i a e i h h e m t t e e s o m m a s w o o e s s t t a s a s a h o a w W h w W n w e o h H W QA-SRL 1.0 • Small dataset • Trained annotators • Only explored sub-problems [He et al. 2015]

Goal A high-quality, large-scale parser for QA-SRL

In 1950 Alan M. Turing published "Computing machinery and intelligence" in Mind, in which he proposed that machines could be tested for intelligence using questions and answers. When was something published? In 1950 Who published something? Alan M. Turing published “Computing Machinery What was published? and Intelligence” Where was something published? in Mind When did someone propose something? In 1950 Who proposed something? Alan M. Turing proposed that machines could be What did someone propose? tested for intelligent using questions and answers “Computing Machinery What did someone propose something in? and Intelligence” What can be tested? machines tested What can something be tested for? intelligence using questions and How can something be tested? answers What was being used? questions and answers using Why was something being used? tested for intelligence

Challenges 1. Scale up QA-SRL data annotation

Challenges 1. Scale up QA-SRL data annotation

Challenges 1. Scale up QA-SRL data annotation 75k sentence dataset in 9 days

Challenges 1. Scale up QA-SRL data annotation 2. Train a QA-SRL Parser Where didn’t someone appear to do something? In the video Who didn’t appear to do something? the perpetrators appeared When did someone appear? never In the video, the look at the camera perpetrators never What didn’t someone appear to do? appeared to look to look at the camera at the camera. Where didn't someone look at something? In the video look Who didn’t look? the perpetrators What didn’t someone look at? the camera

Challenges 1. Scale up QA-SRL data annotation 2. Train a QA-SRL Parser 3. Improve Recall +11% data 
 + 2% Fscore Validate Overgenerate

In 1950 Alan M. Turing published "Computing machinery and intelligence" in Mind, in which he proposed that machines could be tested for intelligence using questions and answers. When was something published? In 1950 Who published something? Alan M. Turing published “Computing Machinery What was published? and Intelligence” Where was something published? in Mind When did someone propose something? In 1950 Who proposed something? Alan M. Turing proposed that machines could be What did someone propose? tested for intelligent using questions and answers “Computing Machinery What did someone propose something in? and Intelligence” What can be tested? machines tested What can something be tested for? intelligence using questions and How can something be tested? answers What was being used? questions and answers using Why was something being used? tested for intelligence

Large-scale QA-SRL Parsing 1. Scale up QA-SRL data annotation 2. Train a QA-SRL Parser 3. Improve Recall

Easier Annotation UCCA ~6k sentences 4 Trained Annotators [Abend and Rapaport 2013] Semantic Proto-roles ~7k sentences MTurk [Reisinger et al. 2015] Trained annotators/ Groningen Meaning Bank ~40k sentences GWAP [Basile et al. 2012] QASRL 1.0 ~3k sentences Trained annotators [He et al. 2015] QA-SRL 2.0 75k sentences MTurk

QA-SRL Questions: Wh Aux Subj Verb Obj Prep Obj2 ∅ ∅ ∅ ∅ ∅ ∅ Who did someone stem someone on someone What didn’t something past something to something Where might past participle by When will present from Why … … How

QA-SRL John surreptitiously ate the burrito at 2am. Questions: Wh Aux Subj Verb Obj Prep Obj2 ∅ ∅ ∅ ∅ ∅ ∅ Who did someone stem someone on someone What didn’t something past something to something Where might past participle by When will present from Why … … How

QA-SRL John surreptitiously ate the burrito at 2am. Questions: Who ate something? Wh Aux Subj Verb Obj Prep Obj2 ∅ ∅ ∅ ∅ ∅ ∅ Who did someone stem someone on someone What didn’t something past something to something Where might past participle by When will present from Why … … How

QA-SRL John surreptitiously ate the burrito at 2am. Questions: Who ate something? What did someone eat? Wh Aux Subj Verb Obj Prep Obj2 ∅ ∅ ∅ ∅ ∅ ∅ Who did someone stem someone on someone What didn’t something past something to something Where might past participle by When will present from Why … … How

QA-SRL John surreptitiously ate the burrito at 2am. Questions: Who ate something? What did someone eat? …

QA-SRL John surreptitiously ate the burrito at 2am. Questions: Answers: Who ate something? John What did someone eat? …

QA-SRL John surreptitiously ate the burrito at 2am. Questions: Answers: Who ate something? John What did someone eat? the burrito …

Annotation Pipeline John surreptitiously ate the burrito at 2am x Predicate Identify verbs with POS + heuristics detection

Annotation Pipeline John surreptitiously ate the burrito at 2am x Predicate Identify verbs with POS + heuristics detection One worker writes as many QA-SRL questions as possible, Question annotation and provides the answer

Annotation Pipeline John surreptitiously ate the burrito at 2am x Predicate Identify verbs with POS + heuristics detection One worker writes as many QA-SRL questions as possible, Question annotation and provides the answer 2 workers are shows questions, Validation provide answers or mark as invalid

Question Annotation • E ffi ciency • Recall

Question Annotation • E ffi ciency • Autocomplete

Question Annotation • E ffi ciency • Autocomplete • Recall • Autosuggest

Question Annotation • E ffi ciency • Autocomplete • Recall • Autosuggest • Financial Incentives

Validation Interface

Dataset • 1 annotator provides questions • 2 annotators validate -> 3 spans / question • Question invalid if any annotator marks invalid • Additional 3 validators for small dense dev and test set

Dataset [He et al 2015] This work 3000 sentences 75k sentences Several weeks 9 days ~50c / verb 33c / verb 2.43 questions / verb 2.05 questions / verb

In 1950 Alan M. Turing published "Computing machinery and intelligence" in Mind, in which he proposed that machines could be tested for intelligence using questions and answers. Who published something? Alan M. Turing “Computing Machinery What was published? and Intelligence” published When was something published? In 1950 Where was something published? in Mind Who proposed something? Alan M. Turing that machines could be What did someone propose? tested for intelligent using proposed questions and answers When did someone propose something? In 1950 “Computing Machinery What did someone propose something in? and Intelligence” What can be tested? machines tested What can something be tested for? intelligence using questions and How can something be tested? answers What was being used? questions and answers using Why was something being used? tested for intelligence