RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6064–6081 July 9-14, 2023 ©2023 Association for Computational Linguistics ROBUT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations Yilun Zhao1 Chen Zhao2 Linyong Nan1 Zhenting Qi3 Wenlin Zhang3 Boyu Mi3 Xiangru Tang1 Dragomir Radev1 1Yale University 2 New York University 3 Zhejiang University yilun.zhao@yale.edu cz1285@nyu.edu Abstract Despite significant progress having been made in question answering on tabular data (Table QA), it’s unclear whether, and to what extent existing Table QA models are robust to task- specific perturbations, e.g., replacing key ques- tion entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called ROBUT, which builds upon existing Table QA datasets (WTQ, WIKISQL-WEAK, and SQA) and includes human-annotated adversarial per- turbations in terms of table header, table con- tent, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial exam- ples to enhance training, which significantly im- proves the robustness of Table QA models. Our data and code is publicly available at https: //github.com/yilunzhao/RobuT. 1 Introduction Table QA uses structured table as world knowledge to answer questions. In recent years, Transformer- based models (Yin et al., 2020; Herzig et al., 2020; Yang et al., 2022; Jiang et al., 2022; Liu et al., 2022; Scao et al., 2022) achieve remarkable results on ex- isting Table QA benchmark datasets (Pasupat and Liang, 2015; Zhong et al., 2017; Iyyer et al., 2017). Despite significant progress, state-of-the-art mod- els are only evaluated within the same distribution, which does not provide insight into the model’s robustness against out-of-domain distribution or adversarial data (Suhr et al., 2020), and recent stud- ies (Cho et al., 2018; Zhu et al., 2020; Yang et al., 2022) revealed that existing models are vulnerable to adversarial perturbations. For example, Cho et al. (2018) observed significant performance degrada- tion after a sentence-level question perturbation. Yang et al. (2022) showed that state-of-the-art Table Year Date Pts Runner-up Attendance 2007 10 Oct 2007 3 - 1 Team A 2,431 2008 22 Jun 2008 4 - 0 Team B 7,460 2009 5 July 2009 2 - 1 Team C 8,519 2010 24 Jan 2010 3 - 2 Team D 6,320 2011 6 Nov 2011 1 - 0 Team E 6,914 Year Date Points Second Place Audience Number 2007 10 Oct 2007 3 - 1 Team A 2,431 2008 22 Jun 2008 4 - 0 Team B 7,460 2009 5 July 2009 2 - 1 Team C 8,519 2010 24 Jan 2010 3 - 2 Team D 6,320 2011 6 Nov 2011 1 - 0 Team E 6,914 Year Date Points Venue Runner-up Attendance Referee 2007 10 Oct 2007 3 - 1 Stadium A Team A 2,431 Person A 2008 22 Jun 2008 4 - 0 Stadium B Team …

Content truncated to 500 words