Question Answering over Freebase with Multi-Column Convolutional - PowerPoint PPT Presentation

Question Answering over Freebase with Multi-Column Convolutional Neural Networks Li Dong 1 , Furu Wei 2 , Ming Zhou 2 , Ke Xu 1 1 SKLSDE, Beihang University, Beijing, China 2 Microsoft Research, Beijing, China

Question Answering over Freebase ▪ Freebase ▪ Large-scale knowledge base ▪ A rich resource to answer open-domain questions Question: Answer: when did Avatar release in UK 2009-12-17 ▪ Challenge ▪ natural language questions ~ structured semantics of Freebase ▪ How to bridge the gap?

Mainstream Methods (1/2) ▪ Semantic parsing (Berant et al., 2013; Bao et al., 2014; etc.) ▪ Question Formal Meaning Representation Structured Queries Answer ▪ ▪ Example ▪ Utterance: Which college did Obama go to ▪ Logical form: (and (Type University) (Education BarackObama)) ▪ Denotation: Occidental College, Columbia University ▪ Challenges ▪ Huge search space ▪ Lexical triggers Example is borrowed from the website of SEMPRE

Mainstream Methods (2/2) ▪ Information extraction over knowledge base ▪ 1. Retrieve candidate answers from Freebase ▪ 2. Extract features ▪ 3. Classification / Ranking Correct Answer Correct Answer Ranking score Classifier Candidate Embedding Features Sum of Word Embeddings Candidate Answers Candidate Answers Question Question (Yao and Van Durme, 2014) (Bordes et al., 2014a; 2014b)

Proposed Method ▪ Question answering -> Constraint matching ▪ Answer type, answer path (relation), answer context ▪ Question understanding with convolutional neural networks Answer Ranker Matching Score Type Relation Context Type Relation Context Multi-column Candidate Answers Convolutional Neural Networks Question

Model Overview Score + + Score Layer Dot Product Answer Answer Type Answer Context Path 2009-12-17 value_type datetime film.film_regional_release _date.release_date film.film. m.0gdp17z Max-Pooling Layer release_date_s Avatar type.object.type m.0bth54 film.film_regional_release _date.film_release_region film.film_region film.film.directed_by al_release_date film.film.release United Kingdom _date_s m.07ssc James Cameron m.03_gd film.film_region Convolutional Layer type.object.type type.object.type m.09w09jk al_release_date type.object.type film.film_regional_release people.person film.film_regional_release _date.release_date Shared Word _date.film_release_region Representations <L> when did Avatar release in UK <R> film.producer 2009-12-18 United States value_type of America m.09c7w0 datetime

Model Overview Score Candidate + + Score Layer Dot Product Answer Answer Answer Answer Ans Type Type Answer Answer Context Context C Path Path 2009-12-17 2009-12-17 value_type value_type datetime datetime film.film_regional_release film.film_regional_release _date.release_date _date.release_date film.film. film.film. m.0gdp17z m.0gdp17z Max-Pooling Layer release_date_s release_date_s Avatar Avatar type.object.type type.object.type m.0bth54 m.0bth54 film.film_regional_release film.film_regional_release _date.film_release_region _date.film_release_region film.film_region film.film_region film.film.directed_by irected_by al_release_date al_release_date film.film.release film.film.release United Kingdom United Kingdom _date_s _date_s m.07ssc m.07ssc James Cameron n m.03_gd film.film_region Convolutional Layer type.object.type type.object.type m.09w09jk al_release_date type.object.type film.film_regional_release people.person film.film_regional_release _date.release_date Shared Word _date.film_release_region Representations <L> when did Avatar release in UK <R> film.producer 2009-12-18 United States value_type of America m.09c7w0 datetime

Embedding Candidate Answers ▪ Learn vector representations for candidate answers ▪ (Bordes et al., 2014a; Bordes et al., 2014b) ▪ Answer path ▪ relations between the candidate node and the entity asked in question ▪ 𝑏𝑤𝑕 𝒔 𝟐 , 𝒔 𝟑 , … , 𝒔 𝒐 : average of relation embeddings Candidate ▪ Answer context Answer Answer Answer Type Context Path ▪ Answer type 2009-12-17 value_type datetime film.film_regional_release Asked entity _date.release_date film.film. m.0gdp17z Avatar release_date_s type.object.type m.0bth54 film.film_regional_release _date.film_release_region film.film_region film.film.directed_by al_release_date film.film.release United Kingdom _date_s m.07ssc James Cameron m.03_gd film.film_region type.object.type .object.type

Embedding Candidate Answers ▪ Learn vector representations for candidate answers ▪ (Bordes et al., 2014a; Bordes et al., 2014b) ▪ Answer context ▪ 1-hop entities and relations connected to the answer path ▪ 𝑏𝑤𝑕 𝒅 𝟐 , 𝒅 𝟑 , … , 𝒅 𝒐 : average of context entity and relation embeddings ▪ Answer path Answer Answer ▪ Answer type Answer Type Context Path 2009-12-17 value_type datetime film.film_regional_release _date.release_date film.film. m.0gdp17z Avatar release_date_s type.object.type m.0bth54 film.film_regional_release _date.film_release_region film.film_region film.film.directed_by al_release_date film.film.release United Kingdom _date_s m.07ssc James Cameron m.03_gd film.film_region type.object.type .object.type

Embedding Candidate Answers ▪ Learn vector representations for candidate answers ▪ (Bordes et al., 2014a; Bordes et al., 2014b) ▪ Answer type ▪ common.topic.notable_types, value type (e.g., float, string, datetime) ▪ 𝑏𝑤𝑕 𝒖 𝟐 , 𝒖 𝟑 , … , 𝒖 𝒐 : average of type embeddings ▪ Answer path Answer Answer ▪ Answer context Answer Type Context Path 2009-12-17 value_type datetime film.film_regional_release _date.release_date film.film. m.0gdp17z Avatar release_date_s type.object.type m.0bth54 film.film_regional_release _date.film_release_region film.film_region film.film.directed_by al_release_date film.film.release United Kingdom _date_s m.07ssc James Cameron m.03_gd film.film_region type.object.type .object.type

Model Overview Score + + Score Layer Dot Product Answer Answer Type Answer Context Path 2009-12-17 value_type datetime film.film_regional_release _date.release_date film.film. m.0gdp17z Max-Pooling Layer Max-Pooling Layer release_date_s Avatar type.object.type m.0bth54 film.film_regional_release _date.film_release_region film.film_region film.film.directed_by al_release_date film.film.release United Kingdom _date_s m.07ssc James Cameron m.03_gd film.film_region Convolutional Layer Convolutional Layer type.object.type type.object.type ty m.09w09jk al_release_date type.object.type film.film_regional_release people.pe people.person film.film_regional_release _date.release_date Shared Word Shared Word _date.film_release_region Representations Representations <L> <L> when when did did Avatar release Avatar release in in UK UK <R> <R> film.producer 2009-12-18 United States value_type of America m.09c7w0 datetime

Model Overview Score Score + + + + Score Layer Score Layer Dot Product Dot Product Answer Answer Answer Answer Type Type Answer Answer Context Context Path Path 2009-12-17 value_type date datetime film.film_regional_release _date.release_date film.film. film.film. m.0gdp17z Max-Pooling Layer release_date_s Avatar type.object.type m.0bth54 film.film_regional_release _date.film_release_region film.film_region film.film.directed_by al_release_date film.film.release United Kingdom _date_s m.07ssc James Cameron m.03_gd film.film_region Convolutional Layer type.object.type type.object.type m.09w09jk al_release_date type.object.type film.film_regional_release people.person film.film_regional_release _date.release_date Shared Word _date.film_release_region Representations <L> when did Avatar release in UK <R> film.producer 2009-12-18 United States value_type of America m.09c7w0 datetime

Model Training ▪ Negative instance 𝑏′ is randomly sampled from the set of candidate answers ▪ Hinge loss for (𝑟, 𝑏) and (𝑟, 𝑏′) , where ▪ Objective function ▪ 𝐵 𝑟 : set of correct answers ▪ 𝑆 𝑟 ⊆ 𝐷 𝑟 \A 𝑟 : set of wrong answers ▪ Back-propagation, AdaGrad, max-norm regularization

Inference (During Test) Answer Ranker 4. Compute scores Matching Score Type Relation Context Type Relation Context 3. Compute vector representations Multi-column Convolutional Neural Networks Candidate Answers (2-hop entities/attributes) when did Avatar release in UK 2. Retrieve candidates Avatar 1. Link to entity in Freebase

Inference (During Test) ▪ If there are more than one correct answers ▪ Use the margin 𝑛 in objective function as threshold ▪ Candidates whose scores are not far from the best answer are regarded as predicted results

Question Paraphrases for Multi-Task Learning ▪ Question understanding results of paraphrases should be same ▪ who is the father of A ▪ who is A’s father ▪ So, the vectors of paraphrases computed by neural networks should be similar ▪ Hinge loss ▪ Negative instance is randomly sampled