Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parsers

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11548 - 11562 December 7-11, 2022 ©2022 Association for Computational Linguistics Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parsers Abhijeet Awasthi Ashutosh Sathe Sunita Sarawagi Indian Institute of Technology Bombay, India {awasthi,absathe,sunita}@cse.iitb.ac.in Abstract Text-to-SQL parsers typically struggle with databases unseen during the train time. Adapt- ing parsers to new databases is a challeng- ing problem due to the lack of natural lan- guage queries in the new schemas. We present REFILL, a framework for synthesizing high- quality and textually diverse parallel datasets for adapting a Text-to-SQL parser to a tar- get schema. REFILL learns to retrieve-and- edit text queries from the existing schemas and transfers them to the target schema. We show that retrieving diverse existing text, mask- ing their schema-specific tokens, and refill- ing with tokens relevant to the target schema, leads to significantly more diverse text queries than achievable by standard SQL-to-Text gen- eration methods. Through experiments span- ning multiple databases, we demonstrate that fine-tuning parsers on datasets synthesized us- ing REFILL consistently outperforms the prior data-augmentation methods. 1 Introduction Natural Language interface to Databases (NLIDB) that translate text queries to executable SQLs is a challenging task in the field of Semantic Pars- ing (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Berant et al., 2013). In addition to understanding the natural language and gener- ating an executable output, Text-to-SQL also re- quires the ability to reason over the schema struc- ture of relational databases. Recently, datasets such as Spider (Yu et al., 2018) comprising of parallel (Text,SQL) pairs over hundreds of schemas have been released, and these have been used to train state-of-the-art neural Text-to-SQL models (Wang et al., 2020; Scholak et al., 2021a; Rubin and Berant, 2021; Scholak et al., 2021b; Xu et al., 2021). However, several studies have indepen- dently shown that such Text-to-SQL models fail catastrophically when evaluated on unseen schemas from the real-world databases (Suhr et al., 2020; Lee et al., 2021; Hazoom et al., 2021). Adapting existing parsers to new schemas is challenging due to the lack of parallel data for fine-tuning the parser. Synthesizing parallel data, that is representative of natural human generated queries (Wang et al., 2015; Herzig and Berant, 2019), is a long-standing problem in semantic parsing. Several methods have been proposed for supplementing with syn- thetic data, ranging from grammar-based canoni- cal queries to full-fledged conditional text genera- tion models (Wang et al., 2015; Herzig and Berant, 2019; Zhong et al., 2020; Yang et al., 2021; Zhang et al., 2021; Wang et al., 2021). For Text-to-SQL, data-augmentation methods are primarily based on training an SQL-to-Text model using labeled data from pre-existing schemas, and generating data in the new schemas. We show that the text gener- ated by these methods, while more natural than canonical queries, lacks the rich diversity of natu- ral multi-user queries. Fine-tuning with such data often deteriorates the model performance since the lack of diversity leads to a biased model. …

Content truncated to 500 words