The 29th International Joint Conference on Artificial Intelligence (IJCAI-20) How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context Qian Liu, Bei Chen , Jiaqi Guo, Jian-Guang Lou, Bin Zhou, Dongmei Zhang
Semantic Parsing Semantic parsing aims to translate a natural language sentence into its corresponding executable programming language, which relieves users from the burden of learning techniques behind the programming language. CARS_DATA Id MPG What is id of the car with the max horsepower? Horsepower SELECT Id FROM CARS_DATA CARS_NAMES ORDER BY Horsepower DESC LIMIT 1 MakeId Model Make How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Semantic Parsing in Context Users prefer to interact with systems in a dialogue, where users are allowed to ask context-dependent questions, which arises the task of Semantic Parsing in Context (SPC). CARS_DATA What is id of the car with the max horsepower? Id MPG SELECT Id FROM CARS_DATA Horsepower ORDER BY Horsepower DESC LIMIT 1 CARS_NAMES How about with the max mpg? MakeId Model SELECT Id FROM CARS_DATA Make ORDER BY MPG DESC LIMIT 1 How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Semantic Parsing in Context What is id of the car with the max horsepower? SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1 Recent Questions as Context How about with the max mpg? Precedent SQL as Context SELECT Id FROM CARS_DATA ORDER BY MPG DESC LIMIT 1 Show its Make! SELECT T1.Make FROM CAR_NAMES AS T1 JOIN CARS_DATA AS T2 ON T1.MakeId = T2.Id ORDER BY T2.MPG DESC LIMIT 1 How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Grammar-based Semantic Parser Start → Root, Root → Select Order, Select → Agg, Agg → max Col SELECT Id FROM CARS_DATA Action SQL Tab, Col → Id, Tab → CARS DATA, Order → desc limit Agg, ORDER BY Horsepower Agg → none Col Tab, Col → Horsepower, Tab → CARS DATA DESC LIMIT 1 → → → → Attention Abstract Syntax Tree … → → → … Encoder Encoder Encoder Encoder Encoder … Ques How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Recent Questions as Context Input Recent ℎ questions 𝐲 𝑗−ℎ , 𝐲 𝑗−ℎ+1 , … , 𝐲 𝑗−1 & Current NL question 𝐲 𝑗 Output Corresponding SQL 𝐳 𝑗 C ONCAT T URN [Suhr et al., 2018] G ATE [Zhang et al., 2018] How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Precedent SQL as Context Input Precedent SQL 𝐳 𝑗−1 & Current NL question 𝐲 𝑗 Output Corresponding SQL 𝐳 𝑗 SQL Attn A CTION C OPY [Zhang et al., 2019] T REE C OPY [Suhr et al., 2018] How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Precedent SQL as Context Input Precedent SQL 𝐳 𝑗−1 & Current NL question 𝐲 𝑗 Output Corresponding SQL 𝐳 𝑗 SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1 Large Overlap SELECT Id FROM CARS_DATA ORDER BY MPG DESC LIMIT 1 How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Precedent SQL as Context Input Precedent SQL 𝐳 𝑗−1 & Current NL question 𝐲 𝑗 Output Corresponding SQL 𝐳 𝑗 SQL Attn A CTION C OPY [Zhang et al., 2019] T REE C OPY [Suhr et al., 2018] How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Research Question How Far are We from Effective Context Modeling? 🤕 How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Experiments • Dataset [Complex, Multi-Table, Cross-Domain] • SParC [Yu et al., 2019b] : Train/Dev = 3034/422 dialogues, average turn number is 3.0 • CoSQL [Yu et al., 2019a] : Train/Dev = 2164/292 dialogues, average turn number is 5.2 • Metrics • SQL Exact Set Match: Question Match, Interaction Match, Turn 𝑗 Match C ONCAT model V.S. SOTA methods ? Comparison of different context modeling methods ? Context models for different contextual phenomena ? How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
C ONCAT model V.S. SOTA methods • Our C ONCAT model outperforms baselines by a large margin, achieving new SOTA performances on both datasets. ✓ SyntaxSQL-con [Yu et al., 2018] is analogous to T URN ✓ CD-Seq2Seq [Suhr et al., 2018] is analogous to T URN + T REE C OPY ✓ EditSQL [Zhang et al., 2019] is analogous to T URN + SQL A TTN + A CTION C OPY How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Comparison of Different Methods • None of the 13 methods can be consistently superior to the others. ✓ Recent questions as context ✓ Precedent SQL as context ✓ Combination methods • We need more effective context models. ✓ Simple/naive methods work well (e.g. C ONACT , T URN , and A CTION C OPY ) 13 Context Modeling Methods How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Comparison on Contextual Phenomena What is id of the car with the max horsepower? Ellipsis How about with the max mpg? Coreference Show its Make! Semantically Complete How many cars in total ? How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Comparison on Coreference Bridging Anaphora Show the version number for all templates. What is the smallest value ? Definite Noun Phrases Which country has a head of state named Beatrix? What languages are spoken in that country ? One Anaphora Order the pets by age. How much does each one weigh? Demonstrative Pronoun Which students have pets? Of those , whose last name is smith? Possessive Determiner How many highschoolers are liked by someone else? What are their names? All methods perform relatively well on types: • ✓ Bridging Anaphora ✓ Demonstrative Pronoun ✓ Possessive Determiner ✓ They refer more to the “ Whole ” Antecedent All methods struggle with types: • ✓ Definite Noun Phrases ✓ One Anaphora ✓ They refer more to the “ Partial ” Antecedent How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Comparison on Ellipsis Continuation What are all the flight numbers? Which land in Aberdeen? Explicit What is id of the car with the max horsepower ? How about with the max MPG ? Implicit Find the names of museums opened before 2010. How about after ? Substitution Schema How many losers participated in the Australian Open? Winners ? Operator Who was the last student to register? Who was the first to register? Performance Continuation > Substitution • For Substitution • ✓ Performance Implicit > Explicit ✓ Performance Operator > Schema How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Conclusion How Far are We from Effective Context Modeling? A long way to go • ✓ In the cross-domain setting, existing models are not as effective as expected. What we can do • ✓ incorporating common sense for better pronouns inference ✓ modeling contextual clues in a more explicit manner ✓ debug models in a fine-grained level How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context
Thanks Paper : How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context Code : https://github.com/microsoft/ContextualSP We are hiring interns : di-recruit@microsoft.com
Recommend
More recommend
