Multitask Pretraining with Structured Knowledge for Text-to-SQL Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11067–11083 July 9-14, 2023 ©2023 Association for Computational Linguistics Multitask Pretraining with Structured Knowledge for Text-to-SQL Generation Robert Giaquinto∗, Dejiao Zhang, Benjamin Kleiner, Yang Li Ming Tan, Parminder Bhatia, Ramesh Nallapati, Xiaofei Ma AWS AI Labs {rgiaq,dejiaoz,kleinerb,ylizam, mingtan,parmib,rnallapa,xiaofeim}@amazon.com Abstract Many machine learning-based low-code or no- code applications involve generating code that interacts with structured knowledge. For exam- ple, one of the most studied tasks in this area is generating SQL code from a natural language statement. Prior work shows that incorporating context information from the database schema, such as table and column names, is beneficial to model performance on this task. In this work we present a large pretraining dataset and strat- egy for learning representations of text, tables, and SQL code that leverages the entire con- text of the problem. Specifically, we build on existing encoder-decoder architecture by intro- ducing a multitask pretraining framework that complements the unique attributes of our di- verse pretraining data. Our work represents the first study on large-scale pretraining of encoder- decoder models for interacting with structured knowledge, and offers a new state-of-the-art foundation model in text-to-SQL generation. We validate our approach with experiments on two SQL tasks, showing improvement over ex- isting methods, including a 1.7 and 2.2 percent- age point improvement over prior state-of-the- arts on Spider and CoSQL. 1 Introduction Tables, relational databases, and other forms of structured knowledge (SK) encompass a massive amount of data across a wide range of applications. Extracting insights held in such data often requires proficiency in query languages like SQL, making it only accessible to the minority of people with the technical skills. A natural language interface, however, would expand access to these information exponentially. Likewise, querying via natural lan- guage allows users quickly hone in on an answer to their particular question, rather than visually scanning dense tables where the majority of the in- formation is irrelevant to the user. To that end, we ∗Corresponding author. explore pretraining techniques for large language models that focus on the challenging interplay be- tween structured and unstructured knowledge, and target a variety of downstream text-to-SQL tasks. Recently there have been significant advance- ments in learning representations for tables (Yin et al., 2020; Herzig et al., 2020; Eisenschlos et al., 2020; Liu et al., 2022; Wang et al., 2021c; Yu et al., 2021; Cheng et al., 2022; Dong et al., 2022), which advanced the state-of-the-art in a range of table-to- text tasks, like table question-answering (Nan et al., 2022; Chen et al., 2021), fact verification (Chen et al., 2020; Aly et al., 2021), data-to-text (Parikh et al., 2020; Nan et al., 2021), and semantic pars- ing (Yu et al., 2019b; Zhong et al., 2017). While better table understanding benefits a range of tasks, pretraining focused on text-to-SQL has thus far re- ceived less attention. Pretrained encoders, such as TaBERT and TAPAS (Yu et al., 2021; Yin et al., 2020; Herzig et …

Content truncated to 500 words