Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372–8388 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8372 Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing Alane Suhr‡∗, Ming-Wei Chang†, Peter Shaw†, and Kenton Lee† ‡ Cornell University Department of Computer Science and Cornell Tech New York, NY 10044 suhr@cs.cornell.edu †Google Research {mingweichang, petershaw, kentonl}@google.com Abstract We study the task of cross-database seman- tic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen dur- ing training. Recently, several datasets, includ- ing Spider, were proposed to support devel- opment of XSP systems. We propose a chal- lenging evaluation setup for cross-database se- mantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic pars- ing, demonstrating the need to use and develop diverse training and evaluation datasets. 1 Introduction Semantic parsing is the task of mapping natural language utterances to formal meaning represen- tations, and has been studied in tasks including instruction following, evaluating sentence mean- ing, and building interfaces to knowledge bases. In this paper, we focus on the task of mapping from natural language utterances to SQL queries exe- cutable in a database. Most prior work in mapping from natural language to SQL queries train and test the system on a single database. We refer to this setup as single-database semantic parsing (SSP). Well-studied datasets used in the SSP setting in- clude GeoQuery (Zelle and Mooney, 1996) and ATIS (Hemphill et al., 1990; Dahl et al., 1994). However, semantic parsing systems should be able to generalize to new domains and databases, ∗Work done during an internship at Google. Advising (Finegan-Dollak et al., 2018) NL: For EECS 478, how many credits is it? SQL: select distinct credits from course where department =‘EECS’ and number = 478; GeoQuery (Zelle and Mooney, 1996) NL: How many people live in mississippi? SQL: select population from state where state name = ‘mississippi’; NL:ATIS (Hemphill et al., 1990; Dahl et al., 1994) Flights from Phoenix to Milwaukee SQL: select distinct T1.flight id from airport service as T2, airport service as T3, city as T4, city as T5, flight as T1 where T4.city code = T2.city code and T4.city name = ‘Phoenix’ and T5.city code = T3.city code and T5.city name = ‘Milwaukee’ and T1.from airport = T2.airport code and T1.to airport = T3.airport code; Spider (Yu et al., 2018) NL: List the emails of the professionals who live in the state of Hawaii or the state of Wisconsin. SQL: select email address from professionals where state = ‘Hawaii’ or state = ‘Wisconsin’; Figure 1: Examples of generalization …

Content truncated to 500 words