Towards Robustness of Text-to-SQL Models against Synonym Substitution

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2505–2515 August 1–6, 2021. ©2021 Association for Computational Linguistics 2505 Towards Robustness of Text-to-SQL Models against Synonym Substitution Yujian Gan1 Xinyun Chen2 Qiuping Huang3 Matthew Purver1 John R. Woodward1 Jinxia Xie4 Pengsheng Huang5 1Queen Mary University of London 2UC Berkeley 3Nanning Central Sub-branch of the People’s Bank of China 4Guangxi University of Finance and Economics 5Beijing MeiyiLab Co.,Ltd. {y.gan,m.purver,j.woodward}@qmul.ac.uk xinyun.chen@berkeley.edu qiuping h@foxmail.com jinxia xie@hotmail.com huangpengsheng@pku.edu.cn Abstract Recently, there has been significant progress in studying neural networks to translate text de- scriptions into SQL queries. Despite achiev- ing good performance on some public bench- marks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the mod- els vulnerable to attacks that break the schema linking mechanism. In this work, we investi- gate the robustness of text-to-SQL models to synonym substitution. In particular, we in- troduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to- SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question para- phrases. We observe that the accuracy dramat- ically drops by eliminating such explicit cor- respondence between NL questions and table schemas, even if the synonyms are not adver- sarially selected to conduct worst-case adver- sarial attacks 1. Finally, we present two cate- gories of approaches to improve the model ro- bustness. The first category of approaches uti- lizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective. 2 1 Introduction Neural networks have become the defacto approach for various natural language processing tasks, in- 1Following the prior work on adversarial learning, worst- case adversarial attacks mean adversarial examples generated by attacking specific models. 2Our code and dataset is available at https://github.com/ygan/Spider-Syn What is the type of the file named "David CV"? What is the type of the document named "David CV"? "document", "users", …… SELECT document_type FROM documents …… Spider Question: Spider-Syn Question: Schema Annotations: SQL: What is the average power for all automobiles produced before 1980? What is the average horsepower for all cars produced before 1980? "horsepower", "cars data", …… SELECT avg(horsepower) FROM CARS_DATA …… Spider Question: Spider-Syn Question: Schema Annotations: SQL: different different modified to modified to modified to different Figure 1: Sample Spider questions that include the same tokens as the table schema annotations, and such questions constitute the majority of the Spider bench- mark. In our Spider-Syn benchmark, we replace some schema words in the NL question with their synonyms, without changing the SQL query to synthesize. cluding text-to-SQL translation. Various bench- marks have been proposed for this task, including earlier small-scale single-domain datasets …

Content truncated to 500 words