Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8926–8931 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8926 Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization Yujian Gan1 Xinyun Chen2 Matthew Purver1,3 1Queen Mary University of London 2UC Berkeley 3Jožef Stefan Institute {y.gan,m.purver}@qmul.ac.uk xinyun.chen@berkeley.edu Abstract Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero- shot cross-domain setting. Despite achiev- ing good performance on some public bench- marks, we observe that existing text-to-SQL models do not generalize when facing do- main knowledge that does not frequently ap- pear in the training data, which may render the worse prediction performance for unseen domains. In this work, we investigate the ro- bustness of text-to-SQL models when the ques- tions require rarely observed domain knowl- edge. In particular, we define five types of domain knowledge and introduce Spider-DK (DK is the abbreviation of domain knowledge), a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-DK are selected from Spi- der, and we modify some samples by adding domain knowledge that reflects real-world question paraphrases. We demonstrate that the prediction accuracy dramatically drops on samples that require such domain knowledge, even if the domain knowledge appears in the training set, and the model provides the correct predictions for related training samples. 1 1 Introduction Research on cross-domain text-to-SQL bench- marks has led to numerous advances. Recent works (Zhao et al., 2021; Rubin and Berant, 2021; Hui et al., 2021) have achieved over 70% accuracy on Spider benchmark (Yu et al., 2018) and over 90% accuracy on WikiSQL benchmark (Zhong et al., 2017), which seems to suggest that existing models already solved most problems in this field. How- ever, the follow-up studies from Deng et al. (2021); Gan et al. (2021); Suhr et al. (2020); Shaw et al. (2021); Oren et al. (2020); Keysers et al. (2020) 1Our dataset is available at https://github.com/ygan/Spider- DK. show that the generalization performance is much worse in more challenging scenarios. For example, Deng et al. (2021) investigate the cases when the explicit mentions of database columns are removed from the question. Similarly, (Gan et al., 2021) ob- serve that the model accuracy dramatically drops by replacing schema-related words with some syn- onyms. On the other hand, Suhr et al. (2020) find that the generalization to other databases is much worse, due to the distribution shift of both questions and SQL queries. These papers introduce important challenges for improving the generalization perfor- mance, i.e., the model trained on a cross-domain text-to-SQL dataset (e.g., Spider (Yu et al., 2018)) does not generalize to a new external database. However, the performance degradation is somehow expected for the following reasons. First, remov- ing the explicit mentions breaks the assumptions that make the schema linking effective. Second, SQL queries in other databases could come from a different distribution; e.g., according to the hard- ness criteria defined by Spider benchmark, over 40% Spider SQL queries are …

Content truncated to 500 words