Fact Checking with Insufficient Evidence

Fact Checking with Insufficient Evidence Pepa Atanasova Jakob Grue Simonsen Christina Lioma Isabelle Augenstein Department of Computer Science, University of Copenhagen, Denmark {pepa, simonsen, c.lioma, augenstein}@di.ku.dk Abstract Automating the fact checking (FC) process relies on information obtained from external sources. In this work, we posit that it is crucial for FC models to make veracity predictions only when there is sufficient evidence and oth- erwise indicate when it is not enough. To this end, we are the first to study what information FC models consider sufficient by introducing a novel task and advancing it with three main contributions. First, we conduct an in-depth empirical analysis of the task with a new fluency-preserving method for omitting infor- mation from the evidence at the constituent and sentence level. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with dif- ferent Transformer architectures and three FC datasets. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset, Suffi- cientFacts1, for FC with omitted evidence. We find that models are least successful in detect- ing missing evidence when adverbial modifiers are omitted (21% accuracy), whereas it is easi- est for omitted date modifiers (63% accuracy). Finally, we propose a novel data augmenta- tion strategy for contrastive self-learning of missing evidence by employing the proposed omission method combined with tri-training. It improves performance for Evidence Suffi- ciency Prediction by up to 17.8 F1 score, which in turn improves FC performance by up to 2.6 F1 score. 1 Introduction Computational fact checking approaches typically use deep learning models to predict the veracity of a claim given background knowledge (Thorne et al., 2018; Leippold and Diggelmann, 2020; 1We make the SufficientFacts dataset and the code for the experiments publicly available both on https:// huggingface.co/datasets/copenlu/sufficient facts and https://github.com/copenlu/sufficient facts. Augenstein, 2021). However, the necessary evi- dence is not always available, either due to incom- plete knowledge sources, or because the claim has newly emerged and the relevant facts are not documented yet. In such cases, FC models should indicate that the information available is insuffi- cient to predict the label, as opposed to making a prediction informed by spurious correlations. Prior work shows that FC models can some- times predict the correct veracity based on just the claim, ignoring the evidence, and that they can overly rely on features such as the word over- lap between the evidence and the claim (Schuster et al., 2019, 2021), leading to biased predictions. However, there are no previous studies on what evidence a FC model considers to be enough for predicting a veracity label. To this end, this work introduces the novel task of Evidence Suffi- ciency Prediction illustrated in Figure 1, which we define as the task of identifying what in- formation is sufficient for making a veracity prediction. This task is related to FC and can op- erate on instances and models from FC datasets, but is focused on evaluating the capability of mod- …

Content truncated to 500 words