Addressing Limitations of Encoder-Decoder Based Approach to Text-to-SQL

Proceedings of the 29th International Conference on Computational Linguistics, pages 1593–1603 October 12–17, 2022. 1593 Addressing Limitations of Encoder-Decoder Based Approach to Text-to-SQL Octavian Popescu IBM Research o.popescu@us.ibm.com Irene Manotas IBM Research irene.Manotas@ibm.com Ngoc Phuoc An Vo IBM Research ngoc.phuoc.an.vo@ibm.com Hangu Yeo IBM Research hangu@us.ibm.com Elahe Khorasani IBM Research elkh@ibm.com Vadim Sheinin IBM Research vadims@us.ibm.com Abstract Most attempts on Text-to-SQL task using encoder-decoder approach show a big problem of dramatic decline in performance for new databases. Models trained on Spider dataset, despite achieving 75% accuracy on Spider de- velopment or test sets, show a huge decline be- low 20% accuracy for databases not in Spider. We present a system that combines automated training-data augmentation and ensemble tech- nique. We achieve double-digit percentage im- provement for databases that are not part of the Spider corpus. 1 Introduction Text-to-SQL is a task to automatically translate a given a natural language question into an SQL formula for a specific database (DB). In 2018, Spider dataset (Tao et al., 2018) was released as a large-scale complex and cross-domain semantic parsing and Text-to-SQL dataset. Since then numerous models have been developed and evaluated using Spider dataset. Although there is a large increase in accuracy, growing from 20% to 70 + % between 2019 and 2021 of top ranked systems, later studies (Suhr et al., 2020a; Shi et al., 2020; Zhong et al., 2020), show that the accuracy of these models drops significantly, in some cases well below 20%, for databases outside Spider dataset. In this paper, we present various experiments for Text-to-SQL task using DB schemas that are inside and outside of Spider dataset. We identify systematic types of errors and analyze their possi- ble causes. From our error analysis, we propose solutions to address the limitations of the underly- ing encoder-decoder architecture and of the Spider dataset. We use training data augmentation tech- nique in order to create a series of models for the same DB schema. By coupling each model SQL re- sult with the information extracted from the query at the NLP step, we interpolate a final SQL formula via Ensemble technique. The system achieves very good results compared to its individual components and can be run on any new DB schema, in or out- side of the Spider dataset, even when there exists a set of very limited number of training examples, on the order of several tens. This paper is organized as follows. In Section 2, we review studies that defined a "standard" encoder- decoder architecture for Text-to-SQL. In Section 3, we define the problem of encoder-decoder ap- proach based on declined performances on new DB schemas and new types of questions, that leads to the challenge of achieving good accuracy. Next, we describe our system in Section 5. The experiment results in Section 6 and conclusions in Section 7. 2 Related Work The Text-to-SQL task is not a new task in NLP and there are many references on its complexity and achievements obtained recently (Navid et al., 2017; Popescu …

Content truncated to 500 words