Strong and Light Baseline Models for Fact-Checking Joint Inference

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4824–4830 August 1–6, 2021. ©2021 Association for Computational Linguistics 4824 Strong and Light Baseline Models for Fact-Checking Joint Inference Kateryna Tymoshenko DISI, University of Trento 38123 Povo (TN), Italy kateryna.tymoshenko@unitn.it Alessandro Moschitti∗ Amazon Manhattan Beach, CA 90266, USA amosch@amazon.com Abstract How to combine several pieces of evidence to verify a claim is an interesting semantic task. Very complex methods have been proposed, combining different evidence vectors using an evidence interaction graph. In this paper, we show that in case of inference based on trans- former models, two effective approaches use either (i) a simple application of max pool- ing over the Transformer evidence vectors; or (ii) computing a weighted sum of the evidence vectors. Our experiments on the FEVER claim verification task show that the methods above achieve the state of the art, constituting strong baseline for much more computationally com- plex methods. 1 Introduction Automatic Fact Checking is quickly gaining at- tention of the NLP and AI communities. The FEVER.ai Fact Extraction and Verification Shared Task (Thorne et al., 2018) provides a benchmark for evaluating fact-checking systems. In FEVER, given a claim, C, and a collection of approximately five million Wikipedia pages, W, the task is to predict whether C is supported (SUP) or refuted (REF) by W, or whether there is not enough infor- mation (NEI) in W to support or refute C. If C is classified as SUP or REF, the respective evidence should be provided. Tab. 1 shows a FEVER claim and the gold-standard evidence refuting it. The overall task is complex, as one needs to re- trieve the documents that contain the evidence (doc- ument retrieval, DocIR), select relevant evidence (evidence selection, ES) and label the claim given the evidence (evidence reasoning, ER), which is the focus of our work. Formally, given a claim, C, and a list top K evidence sentences, (E1, ..., EK), retrieved with DocIR and selected by ES respec- ∗Professor at the University of Trento. Claim Evidence Coeliac disease is not treated by maintaining a gluten-free diet. (REF) [(Coeliac disease, “The only known effective treatment is a strict lifelong gluten-free diet....”)] Table 1: FEVER data examples tively, the ER task is to predict the claim label (SUP/REF/NEI). There can be multiple inter-dependent evidence sentences per claim, thus joint modeling them al- lows for taking multiple clues into account, thus intuitively improving system accuracy. Indeed, in- dividual sentences may not constitute standalone evidence, but they can contain several clues, which, together, can support or refute the claim. For ex- ample, Sentence 8 of the Gluten-free diet Wikipedia page, “..gluten-free diet is demonstrated as an effective treatment, but several studies show that about 79 % ... an incomplete recovery of the small bowel...”, which is not listed as ground truth evidence for the claim, still supports the REF sig- nal. Given the above intuition, recent state-of-the-art (SOTA) approaches (Zhou et al., 2019; Ye et al., 2020; Liu et al., 2020; Zhong et al., …

Content truncated to 500 words