Counterfactual Debiasing for Fact Verification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6777–6789 July 9-14, 2023 ©2023 Association for Computational Linguistics Counterfactual Debiasing for Fact Verification Weizhi Xu1,2∗ Qiang Liu1,2∗ Shu Wu1,2† Liang Wang1,2 1Center for Research on Intelligent Perception and Computing, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences 2School of Artificial Intelligence, University of Chinese Academy of Sciences weizhi.xu@cripac.ia.ac.cn, {qiang.liu,shu.wu,wangliang}@nlpr.ia.ac.cn Abstract Fact verification aims to automatically judge the veracity of a claim according to several pieces of evidence. Due to the manual con- struction of datasets, spurious correlations be- tween claim patterns and its veracity (i.e., bi- ases) inevitably exist. Recent studies show that models usually learn such biases instead of understanding the semantic relationship be- tween the claim and evidence. Existing debi- asing works can be roughly divided into data- augmentation-based and weight-regularization- based pipeline, where the former is inflexible and the latter relies on the uncertain output on the training stage. Unlike previous works, we propose a novel method from a counterfactual view, namely CLEVER, which is augmentation- free and mitigates biases on the inference stage. Specifically, we train a claim-evidence fusion model and a claim-only model independently. Then, we obtain the final prediction via sub- tracting output of the claim-only model from output of the claim-evidence fusion model, which counteracts biases in two outputs so that the unbiased part is highlighted. Compre- hensive experiments on several datasets have demonstrated the effectiveness of CLEVER. 1 Introduction Unverified claims have been prevalent online with the dramatic increase of information, which poses a threat to public security over various domains, e.g., public health (Naeem and Bhatti, 2020), poli- tics (Allcott and Gentzkow, 2017), and economics (Kogan et al., 2019). Therefore, fact verification, which aims to automatically predict the veracity of claims based on several collected evidence, has attracted lots of research interests (Liu et al., 2020; Zhong et al., 2020; Vo and Lee, 2021; Jin et al., 2022; Yang et al., 2022). *Equal contribution. †To whom correspondence should be addressed. Existing fact-checking datasets inevitably in- volve some biases since they are manually col- lected. For example, Schuster et al. (2019) discover that negation words in claims are highly-correlated with the label ‘REFUTES’ in the FEVER dataset (Thorne et al., 2018). Such biases may mislead models to explore the spurious correlation between claim patterns and its label without looking into the evidence. In consequence, though models achieve promising performance on biased datasets, they suffer from obvious performance decline on out- of-domain unbiased datasets and are vulnerable to adversarial attacks (Thorne et al., 2019). To alleviate the aforementioned problems, sev- eral debiasing methods have been proposed, which can be mainly grouped into two categories. The first pipeline is based on data augmentation, which utilizes manually-designed schemes, such as word swapping (Wei and Zou, 2019) and span replace- ment (Lee et al., 2021) to generate additional data for training. However, these methods heavily rely on the quality of augmented …

Content truncated to 500 words