Scott Wen-tau Yih # , What are the names of Obamas daughters? Q: - PowerPoint PPT Presentation

Scott Wen-tau Yih # ,

“ What are the names of Obama’s daughters? ”

Q: Who won the best actor Oscar in 1973? S 1 : Jack Lemmon was awarded the Best Actor Oscar for Save the Tiger (1973). S 2 : Academy award winner Kevin Spacey said that Jack Lemmon is remembered as always making time for others.

• Question distribution • Candidate selection bias Q: How did Seminole war end? A: Ultimately, the Spanish Crown ceded the colony to United States rule. • Excluding questions that have no correct answers

• Question distribution • Candidate selection • Including questions that have no correct answers

• Introduction • WikiQA Dataset • Data construction and annotation • Data statistics

“ Who wrote second Corinthians? ”

• Step 1: Does the short paragraph answer the question? Question: Who wrote second Corinthians? Second Epistle to the Corinthians The Second Epistle to the Corinthians, often referred to as Second Corinthians (and written as 2 Corinthians), is the eighth book of the New Testament of the Bible. Paul the Apostle and “Timothy our brother” wrote this epistle to “the church of God which is at Corinth, with all the saints which are in all Achaia”.

• Step 2: Check all the sentences that can answer the question in isolation (assuming coreferences and pronouns have been resolved correctly) Question: Who wrote second Corinthians? Second Epistle to the Corinthians q The Second Epistle to the Corinthians, often referred to as Second Corinthians (and written as 2 Corinthians), is the eighth book of the New Testament of the Bible. q Paul the Apostle and “Timothy our brother” wrote this epistle to “the church of God which is at Corinth, with all the saints which are in all Achaia”.

3,047 3,500 3,000 2,500 13.4x 2,000 1,500 1,000 227 500 - # of questions QASent WikiQA

35,000 29,258 30,000 3.5x 25,000 20,000 15,000 8,478 10,000 3,047 5,000 227 - # of questions # of sentences QASent WikiQA

QASent WikiQA Abbr. Desc. Abbr. Entity Entity 1% 7% 1% 14% 16% Desc. Locatio 35% Human n Locatio 29% 12% n 16% Numeri Numeri c Human c 22% 16% 31%

• Introduction • WikiQA Dataset • Experiments • Baseline systems • Evaluation on answer sentence selection • Evaluation metric & results on answer triggering

• Word matching count (Wd-Cnt) • # non-stopwords in Q that also occur in S • Latent word alignment (Wd-Algn) [Yih+ ACL-13] 𝑟 : What is the fastest car in the world? 𝑔(𝑟 , 𝑡) = ​𝜄↑𝑈 Φ( ℎ ) 𝒊 𝑡 : The Jaguar XJ220 is the dearest, fastest and most sought after car on the planet. • Convolutional NN (CNN) [Yu+ DLWorkshop-14] • Convolutional NN & Wd-Cnt (CNN-Cnt)

0.9 0.7617 0.7633 Mean Reciprocal Rank (MRR) 0.8 0.6662 0.7 0.623 0.6 0.5 0.4 0.3 0.2 0.1 0 QASent Wd-Cnt Wd-Algn CNN CNN-Cnt

0.9 0.7617 0.7633 Mean Reciprocal Rank (MRR) 0.8 0.6662 0.6652 0.7 0.6281 0.623 0.6086 0.6 0.4924 0.5 0.4 0.3 0.2 0.1 0 QASent WikiQA* Wd-Cnt Wd-Algn CNN CNN-Cnt

• Data – All questions in WikiQA are included in this task

33 32.5 Question-level F1 score 32 31.5 31 30.61 30.5 30 29.5 29 28.5 28 CNN-Cnt +Q-Length +S-Length +Q-Class +All

33 32.5 32.17 Question-level F1 score 32 31.5 31 30.61 30.5 30 29.5 29 28.5 28 CNN-Cnt +Q-Length +S-Length +Q-Class +All

33 32.5 32.17 Question-level F1 score 32 31.64 31.5 30.92 31 30.61 30.34 30.5 30 29.5 29 28.5 28 CNN-Cnt +Q-Length +S-Length +Q-Class +All

http://aka.ms/WikiQA • Includes “answer phrases” labeled by authors