Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2007 - 2022 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation Xinyu Pi1∗, Bing Wang2∗, Yan Gao3, Jiaqi Guo4, Zhoujun Li2, Jian-Guang Lou3 1 University of Illinois Urbana-Champaign, Urbana, USA, 2 State Key Lab of Software Development Environment, Beihang University 3 Microsoft Research Asia 4 Xi’an Jiaotong University, Xi’an, China xinyupi2@illinois.edu; {bingwang, lizj}@buaa.edu.cn jasperguo2013@stu.xjtu.edu.cn; {yan.gao, jlou}@microsoft.com Abstract The robustness of Text-to-SQL parsers against adversarial perturbations plays a crucial role in delivering highly reliable applications. Previ- ous studies along this line primarily focused on perturbations in the natural language ques- tion side, neglecting the variability of tables. Motivated by this, we propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm to measure the robustness of Text- to-SQL models. Following this proposition, we curate ADVETA, the first robustness eval- uation benchmark featuring natural and real- istic ATPs. All tested state-of-the-art mod- els experience dramatic performance drops on ADVETA, revealing models’ vulnerability in real-world practices. To defend against ATP, we build a systematic adversarial training ex- ample generation framework tailored for bet- ter contextualization of tabular data. Experi- ments show that our approach not only brings the best robustness improvement against table- side perturbations but also substantially em- powers models against NL-side perturbations. We release our benchmark and code at: https://github.com/microsoft/ContextualSP. 1 Introduction The goal of Text-to-SQL is to generate an exe- cutable SQL query given a natural language (NL) question and corresponding tables as inputs. By helping non-experts interact with ever-growing databases, this task has many potential applications in real life, thereby receiving considerable interest from both industry and academia (Li and Jagadish, 2016; Zhong et al., 2017; Affolter et al., 2019). Recently, existing Text-to-SQL parsers have been found vulnerable to perturbations in NL ques- tions (Gan et al., 2021; Zeng et al., 2020; Deng et al., 2021). For example, Deng et al. (2021) re- moved the explicit mentions of database items in a ∗Equal contributions during the internship at Microsoft Research Asia. Student Name Citizenship Score Semester A Country X 92 Fall B Country Y 90 Spring A Country X 89 B Country Y 85 Fall C Country Z 97 Spring Original Table List names and citizenships of students who achieved top 3 scores. SELECT Student Name, Citizenship FROM Student ORDER BY Score desc LIMIT 3 SELECT Student Name FROM Student ORDER BY Score desc LIMIT 3 (Missing Nationality) SELECT Student Name, Instructor Name, Citizenship FROM Student ORDER BY Grade desc LIMIT 3 Student Name Nationality Score Semester A Country X 92 Fall B Country Y 90 Spring A Country X 89 Spring B Country Y 85 Fall C Country Z 97 Spring RPL Perturbed Table Student Name Citizenship Score Semester Instructor Name Grade A Country X 92 Fall D 6 B Country Y 90 Spring E 6 A Country X 89 Spring E 6 B Country Y 85 Fall …

Content truncated to 500 words