SParC: Cross-Domain Semantic Parsing in Context

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4511–4523 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4511 SParC: Cross-Domain Semantic Parsing in Context Tao Yu† Rui Zhang† Michihiro Yasunaga† Yi Chern Tan† Xi Victoria Lin¶ Suyi Li† Heyang Er† Irene Li† Bo Pang† Tao Chen† Emily Ji† Shreya Dixit† David Proctor† Sungrok Shim† Jonathan Kraft† Vincent Zhang† Caiming Xiong¶ Richard Socher¶ Dragomir Radev† †Department of Computer Science, Yale University ¶Salesforce Research {tao.yu, r.zhang, michihiro.yasunaga, dragomir.radev}@yale.edu {xilin, cxiong, rsocher}@salesforce.com Abstract We present SParC, a dataset for cross-domain Semantic Parsing in Context. It consists of 4,298 coherent question sequences (12k+ indi- vidual questions annotated with SQL queries), obtained from controlled user interactions with 200 complex databases over 138 do- mains. We provide an in-depth analysis of SParC and show that it introduces new chal- lenges compared to existing datasets. SParC (1) demonstrates complex contextual depen- dencies, (2) has greater semantic diversity, and (3) requires generalization to new domains due to its cross-domain nature and the un- seen databases at test time. We experiment with two state-of-the-art text-to-SQL mod- els adapted to the context-dependent, cross- domain setup. The best model obtains an exact set match accuracy of 20.2% over all questions and less than 10% over all inter- action sequences, indicating that the cross- domain setting and the contextual phenomena of the dataset present significant challenges for future research. The dataset, baselines, and leaderboard are released at https:// yale-lily.github.io/sparc. 1 Introduction Querying a relational database is often challeng- ing and a natural language interface has long been regarded by many as the most powerful database interface (Popescu et al., 2003; Bertomeu et al., 2006; Li and Jagadish, 2014). The problem of mapping a natural language utterance into exe- cutable SQL queries (text-to-SQL) has attracted increasing attention from the semantic parsing community by virtue of a continuous effort of dataset creation (Zelle and Mooney, 1996; Iyyer et al., 2017; Zhong et al., 2017; Finegan-Dollak et al., 2018; Yu et al., 2018a) and the modeling innovation that follows it (Xu et al., 2017; Wang et al., 2018; Yu et al., 2018b; Shi et al., 2018). D1 :Database about student dormitory containing 5 tables. C1 :Find the first and last names of the students who are living in the dorms that have a TV Lounge as an amenity. Q1 : ​How many dorms have a TV Lounge? S1 : ​SELECT​ ​COUNT​(*) ​FROM​ dorm ​AS​ T1 ​JOIN​ has_amenity ​AS​ T2 ​ON​ T1.dormid = T2.dormid ​JOIN​ dorm_amenity ​AS​ T3 ​ON​ T2.amenid = T3.amenid ​WHERE T3.amenity_name = 'TV Lounge' Q2 : ​What is the total capacity of these dorms? S2 : ​SELECT​ ​SUM​(T1.student_capacity) ​FROM​ dorm ​AS​ T1 ​JOIN​ has_amenity ​AS​ T2 ​ON​ T1.dormid = T2.dormid ​JOIN​ dorm_amenity ​AS​ T3 ​ON​ T2.amenid = T3.amenid ​WHERE​ T3.amenity_name = 'TV Lounge' Q3 : ​How many students are living there? S3 : SELECT​ ​COUNT​(*) ​FROM​ student ​AS​ T1 ​JOIN​ lives_in ​AS​ T2 ​ON​ T1.stuid = T2.stuid ​WHERE​ T2.dormid ​IN (​SELECT​ T3.dormid ​FROM​ …

Content truncated to 500 words