DuoRAT: Towards Simpler Text-to-SQL Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1313–1321 June 6–11, 2021. ©2021 Association for Computational Linguistics 1313 DuoRAT: Towards Simpler Text-to-SQL Models Torsten Scholak∗, Raymond Li∗, Dzmitry Bahdanau, Harm de Vries, Chris Pal Element AI, a ServiceNow company Abstract Recent neural text-to-SQL models can ef- fectively translate natural language questions to corresponding SQL queries on unseen databases. Working mostly on the Spider dataset, researchers have proposed increas- ingly sophisticated solutions to the problem. Contrary to this trend, in this paper we focus on simplifications. We begin by building Duo- RAT, a re-implementation of the state-of-the- art RAT-SQL model that unlike RAT-SQL is using only relation-aware or vanilla transform- ers as the building blocks. We perform sev- eral ablation experiments using DuoRAT as the baseline model. Our experiments confirm the usefulness of some techniques and point out the redundancy of others, including struc- tural SQL features and features that link the question with the schema1. 1 Introduction Language user interfaces to databases allow non- specialists to retrieve and process information that might otherwise not be easily available to them. Much of the recent research in this area has fo- cused on neural models that can generalize to new relational databases without any human interven- tion. Given a relational database schema (and often also content), such models translate the user’s ques- tion directly into an SQL query (Zhong et al., 2017; Yu et al., 2018a; Bogin et al., 2019). Such cross- database text-to-SQL research was spurred by the introduction of large datasets such as WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018b) that feature utterance-query pairs for hundreds or even thousands of databases. State-of-the-art text-to-SQL models employ many sophisticated techniques. These include, but are not limited to, grammar-constrained models ∗Equal contribution, order was determined by a quantum random number draw. 1Code available at https://github.com/ElementAI/ duorat. and recurrent neural networks with parent feed- ing (Yin and Neubig, 2018), intermediate mean- ing representations (Guo et al., 2019; Suhr et al., 2020), relation-aware attention (Wang et al., 2020), schema linking and table joining heuristics (Guo et al., 2019), slot filling (Choi et al., 2020) and re-ranking (Kelkar et al., 2020) models. The high complexity of these models raises the barrier of entry and can slow down text-to-SQL research. In this work, we attempt to distill the essence of high-performing text-to-SQL systems. We start with a transformer-only reimplementation of the state-of-the-art RAT-SQL model (Wang et al., 2020). Importantly, our resulting DuoRAT model trains three times faster than RAT-SQL. We then systematically study how DuoRAT can be simpli- fied without losing performance. Our ablation study confirms the usefulness of many but not all techniques employed in RAT-SQL. For exam- ple, we show that the benefits of explicit match- ing of question spans with the column or table names (name-based schema linking, NBSL) be- come marginal when a pretrained transformer (De- vlin et al., 2018) is used to jointly encode the ques- …

Content truncated to 500 words