Evidence-based Factual Error Correction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3298–3309 August 1–6, 2021. ©2021 Association for Computational Linguistics 3298 Evidence-based Factual Error Correction James Thorne Department of Computer Science University of Cambridge jt719@cam.ac.uk Andreas Vlachos Department of Computer Science University of Cambridge av308@cam.ac.uk Abstract This paper introduces the task of factual er- ror correction: performing edits to a claim so that the generated rewrite is better supported by evidence. This extends the well-studied task of fact verification by providing a mech- anism to correct written texts that are refuted or only partially supported by evidence. We demonstrate that it is feasible to train fac- tual error correction systems from existing fact checking datasets which only contain labeled claims accompanied by evidence, but not the correction. We achieve this by employing a two-stage distant supervision approach that in- corporates evidence into masked claims when generating corrections. Our approach, based on the T5 transformer and using retrieved ev- idence, achieved better results than existing work which used a pointer copy network and gold evidence, producing accurate factual er- ror corrections for 5x more instances in human evaluation and a .125 increase in SARI score. The evaluation is conducted on a dataset of 65,000 instances based on a recent fact veri- fication shared task and we release it to enable further work on the task.1 1 Introduction Fact verification is the task of predicting whether claims are true or false using evidence. With the availability of a number of resources (Wang, 2017; Karadzhov et al., 2017; Thorne et al., 2018; Au- genstein et al., 2019; Wadden et al., 2020), the task has attracted significant attention and spawned the development of new models, architectures and ap- proaches. With potentially sensitive applications, recent works have focused on building explain- able variants of fact checking (Atanasova et al., 2020; Stammbach and Ash, 2020; Kotonya and Toni, 2020). Exposing the evidence source and 1https://github.com/j6mes/ 2021-acl-factual-error-correction System Outputs Brown recluse spiders do not bite The brown recluse spider's bite sometimes requires medical attention. Input Claim Similar to other recluse spider bites, their bite sometimes requires medical attention. Retrieved Evidence Fact Verification Wikipedia REFUTED Error Correction Information Retrieval Figure 1: Factual Error Correction uses evidence to make corrections to claims, in contrast to fact verifica- tion, which instead classifies the veracity of the claim. decision making process may help the reader un- cover subtle issues that cause automated systems to fail. Additionally, using such evidence to contin- uously update news articles as facts change forms part of the vision outlined by Cohen et al. (2011) for automated newsrooms. In this paper, we propose Factual Error Correc- tion, as an explainable alternative for fact verifica- tion. Rather than merely assigning a truth label, possibly accompanied by evidence, our goal is to rewrite claims so that they are better supported by the retrieved evidence. For example, in Figure 1, a claim that would be REFUTED by the …

Content truncated to 500 words