KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2261–2273 August 1–6, 2021. ©2021 Association for Computational Linguistics 2261 KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers Chia-Hsuan Lee♢ Oleksandr Polozov♠ Matthew Richardson♠ ♢University of Washington ♠Microsoft Research, Redmond chiahlee@uw.edu {polozov,mattri}@microsoft.com Abstract The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application do- mains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical de- ployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evalua- tion tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database do- cumentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of- the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance. 1 Introduction Text-to-SQL parsing is a form of database ques- tion answering (DBQA) that answers a user’s natural-language (NL) question by converting it into a SQL query over a given relational database. It can facilitate NL-based interfaces for arbitrary end- user applications, thereby removing the need for domain-specific UX or learning query languages. As such, DBQA attracted significant attention in academia and industry, with development of super- vised datasets (Yu et al., 2018), large-scale mod- els (Wang et al., 2020b; Zeng et al., 2020), and novel modeling techniques (Yu et al., 2020; Deng et al., 2020). The key challenge of text-to-SQL parsing is zero- shot generalization to unseen domains, i.e. to new database schemas and differently distributed NL questions. Large-scale annotated datasets like Spi- der (Yu et al., 2018) and WikiSQL (Zhong et al., 2017) evaluate cross-domain generalization of text- to-SQL parsers by restricting overlap between train and test domains. Such challenging benchmarks facilitate rapid progress in DBQA. State-of-the-art (SOTA) accuracy on Spider rose from 12.4% to 70.5% in just two years since its release, demonstrat- ing the value of well-chosen evaluation settings. Despite impressive progress in DBQA, deploy- ment of SOTA parsers is still challenging. They often lack robustness necessary to deploy on real- life application domains. While many challenges underlie the gap between SOTA DBQA and its real- life deployment, we identify three specific discrep- ancies. First, Spider and WikiSQL datasets normalize and preprocess database schemas or rely on aca- demic example databases that originate with human- readable schemas (Suhr et al., 2020). In contrast, in- dustrial databases feature abbreviated and obscure naming of table, columns, and data values, often ac- crued from legacy development or migrations. Fig- ure 1 shows a characteristic example. After deploy- ment, text-to-SQL parsers struggle with schema linking to …

Content truncated to 500 words