TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for - PowerPoint PPT Presentation

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer Presenter: Zhuolun Xiang

Background • Question Answering (QA) Formulation • Answer a question 𝑟 given evidences 𝐸 • Dataset of tuples 𝑟 𝑗 , 𝑏 𝑗 , 𝐸 𝑗 𝑗 = 1, … , 𝑜} • 𝑏 𝑗 is a substring of D 𝑗 • Example

Overview • TriviaQA • Over 650K question-answer-evidence triples • First dataset where questions authored by trivia enthusiasts • Evidence documents from Web search and Wiki pages • A high percentage of the questions are challenging • Dataset samples

Dataset Collection • Gather question-answer pairs from 14 trivia websites • Remove short questions • Collect evidence from Web search and Wikipedia • Web search • Pose questions on Bing • Exclude trivia websites • Crawl top 10 results • Wikipedia • Use TAGME to find Wikipedia entities in the question • Add these pages as evidence

Dataset Analysis • Question-answer pairs • Avg length = 14 • Manually analyze 200 sampled questions Property of questions

Dataset Analysis • Question-answer pairs • Avg length = 14 • Manually analyze 200 sampled questions Property of answers

Dataset Analysis • Question-answer pairs • Avg length = 14 • Manually analyze 200 sampled questions • Evidences • 75.4%/79.7% of Web/Wiki evidences contain answers • Human test achieves 75.3/79.6 accuracy on Web/Wiki domains • Answer 40% of questions needs information from multiple sentences

Experiments: Baseline Methods • Random entity baseline (Wiki domain only) • Entities in Wiki pages form candidate answer set • Randomly pick one that not occur in question • Entity classifier • Ranking problem over candidate answers • Function learnt using LambdaMART (Wu et al., 10) • Neural model • Use BiDAF model (Seo et al., 17)

Experiments • Metrics • Exact match(EM) and F1 score • For numerical and freeform answer: single given answer as ground truth • For Wiki entity: use Wiki aliases as well • Setup • Random partition into train(80%)/development(10%)/test(10%)

Experiments • Results • Human baseline: 79.7% on Wiki, 75.4% on web

Conclusion • TriviaQA • 650K question-answer-evidence triples • Questions authored by trivia enthusiasts • Evidence documents from Web search and Wiki pages • Experiments show TriviaQA is a challenging testbed Thanks!

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for - PowerPoint PPT Presentation

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer Presenter: Zhuolun Xiang Background Question Answering (QA) Formulation Answer a

FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large

INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO

Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008

Large-scale production of graphene: Introduction Large-scale production of graphene: what is

Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA

An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of

Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E.

Motivation Large-scale distributed systems becoming more common multiple datacenters, cloud

Large-Scale Electronic Voting Protocols Mike Carpenter Introduction What is meant by large-scale

Internet Topology Generation for Large Scale BGP Simulation Jean-Michel Fourneau - Houssame

E Evolution of NTCIR: l Infrastructure of Large-Scale Infrastructure of Large Scale

Large-Scale Machine Learning at Twitter 2 Large-Scale Machine Learning at Twitter Jimmy Lin and

Large Scale I nternational I Pv6 Pilot Large Scale I nternational I Pv6 Pilot Network (6NET)

INFRASTRUCTURE 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2

A large-scale chemical data integration system Gaia Paolini Pfizer Confidential 1 Large-Scale

Very Large Scale Neighborhoods Weighted Matching Neighborhoods Cyclic Exchange Neighborhoods

Not to Cry Wolf: Distantly Supervised Multitask Learning in Critical Care Patrick Schwab 1

Cosmic Calibration Katrin Heitmann Statistical Challenges for Large-Scale Structure in the Era of

Concretely Efficient La Large-Sc Scale M MPC wi with th Acti tive Securi rity ty (or

A Scalable Scalable Approach Approach A for for Large- -Scale Scale Schema Schema

INTERNET SERVICES 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2

A GLOBALLY LINEARLY CONVERGENT METHOD FOR LARGE-SCALE POINTWISE QUADRATICALLY SUPPORTABLE

LSS Large-Scale Systems Group Tallinn University of Technology LSS Paper Authors Large-Scale

Tanks Later Impacts and Results of Starting Large Scale FSM Michael McWhirter Acknowledgement: