Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3899–3916 June 6–11, 2021. ©2021 Association for Computational Linguistics 3899 Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation Chong Zhang Jieyu Zhao Huan Zhang Kai-Wei Chang Cho-Jui Hsieh Department of Computer Science, UCLA {chongz, jyzhao, kwchang, chohsieh}@cs.ucla.edu, huan@huan-zhang.com Abstract Robustness and counterfactual bias are usually evaluated on a test dataset. However, are these evaluations robust? If the test dataset is perturbed slightly, will the evaluation results keep the same? In this paper, we propose a “double perturbation” framework to uncover model weaknesses beyond the test dataset. The framework first perturbs the test dataset to construct abundant natural sentences similar to the test data, and then diagnoses the prediction change regarding a single-word substitution. We apply this framework to study two perturbation-based approaches that are used to analyze models’ robustness and counterfactual bias in English. (1) For robustness, we focus on synonym substitu- tions and identify vulnerable examples where prediction can be altered. Our proposed attack inattains high success rates (96.0%–99.8%) finding vulnerable examples on both original and robustly trained CNNs and Transformers. (2) For counterfactual bias, we focus on substituting demographic tokens (e.g., gender, race) and measure the shift of the expected prediction among constructed sentences. Our method is able to reveal the hidden model biases not directly shown in the test dataset. Our code is available at https://github.com/chong-z/ nlp-second-order-attack. 1 Introduction Recent studies show that NLP models are vulner- able to adversarial perturbations. A seemingly “invariance transformation” (a.k.a. adversarial per- turbation) such as synonym substitutions (Alzantot et al., 2018; Zang et al., 2020) or syntax-guided paraphrasing (Iyyer et al., 2018; Huang and Chang, 2021) can alter the prediction. To mitigate the model vulnerability, robust training methods have been proposed and shown effective (Miyato et al., 2017; Jia et al., 2019; Huang et al., 2019; Zhou et al., 2020). x0 ="a deep and meaningful film (movie)." Xtest ˜x0 ="a short and moving film (movie)." 73% positive (70% negative) (99% positive) 99% positive perturb Figure 1: A vulnerable example beyond the test dataset. Numbers on the bottom right are the sentiment predic- tions for film and movie. Blue x0 comes from the test dataset and its prediction cannot be altered by the substitution film → movie (robust). Yellow exam- ple ˜x0 is slightly perturbed but remains natural. Its pre- diction can be altered by the substitution (vulnerable). In most studies, model robustness is evaluated based on a given test dataset or synthetic sentences constructed from templates (Ribeiro et al., 2020). Specifically, the robustness of a model is often eval- uated by the ratio of test examples where the model prediction cannot be altered by semantic-invariant perturbation. We refer to this type of evaluations as the first-order robustness evaluation. However, even if a model is first-order robust on an input sen- tence x0, it is possible that the model is not robust on a natural sentence ˜x0 that is …

Content truncated to 500 words