Bridging the Generalization Gap in Text-to-SQL Parsing with Schema Expansion

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5568 - 5578 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Bridging the Generalization Gap in Text-to-SQL Parsing with Schema Expansion Chen Zhao∗ New York University cz1285@nyu.edu Yu Su Microsoft Semantic Machines yusu2@microsoft.com Adam Pauls Microsoft Semantic Machines adam.pauls@microsoft.com Emmanouil Antonios Platanios Microsoft Semantic Machines anthony.platanios@microsoft.com Abstract Text-to-SQL parsers map natural language questions to programs that are executable over tables to generate answers, and are typically evaluated on large-scale datasets like SPIDER (Yu et al., 2018). We argue that existing bench- marks fail to capture a certain out-of-domain generalization problem that is of significant practical importance: matching domain specific phrases to composite operations over columns. To study this problem, we propose a synthetic dataset and a re-purposed train/test split of the SQUALL dataset (Shi et al., 2020) as new benchmarks to quantify domain generalization over column operations. Our results indicate that existing state-of-the-art parsers struggle in these benchmarks. We propose to address this problem by incorporating prior domain knowl- edge by preprocessing table schemas, and de- sign a method that consists of two components: schema expansion and schema pruning. This method can be easily applied to multiple ex- isting base parsers, and we show that it sig- nificantly outperforms baseline parsers on this domain generalization problem, boosting the underlying parsers’ overall performance by up to 13.8% relative accuracy gain (5.1% absolute) on the new SQUALL data split. 1 Introduction Text-to-SQL parsing is the task of translating natural language questions over provided tables to SQL queries which can be executed to produce answers. In recent years, with the availability of large-scale datasets (e.g., Zhong et al., 2017; Yu et al., 2018), neural semantic parsers have witnessed significant success on this task. However, recent work (Suhr et al., 2020; Lee et al., 2021) has suggested that these state-of-the-art parsers are far from successful in terms of out-of-domain generalization in real scenarios, where users may ask questions related to potentially very large tables with the goal of improving their productivity (e.g., while they are viewing or editing a large Excel spreadsheet). ∗ This work was performed during a research internship at Microsoft Semantic Machines. Name Income Armani … Name Earnings Armani … Name Salary Stock Armani … … Name Salary Bonus Armani … … Name Wages Armani … Employee Income Armani … Train Dataset Test Dataset + Question What was Armani’s income last year? Column Matching Column Operations Figure 1: Illustration of two aspects of out-of-domain generalization that are challenging for text-to-SQL parsers. While existing methods partially address the “column matching” issue, they still suffer when it comes to “column operations”. Note that there are more tables on the right to illustrate the fact that there are a variety of settings the parser may run into at test time. In such scenarios, it is common to encounter tables specific to new domains that were not encountered while training a parser. Perhaps the most challenging aspect of …

Content truncated to 500 words