Constrained Fact Verification for FEVER

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7826–7832, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7826 Constrained Fact Verification for FEVER Adithya Pratapa∗, Sai Muralidhar Jayanthi∗, Kavya Nerella∗ Language Technologies Institute Carnegie Mellon University {vpratapa,sjayanth,knerella}@cs.cmu.edu Abstract Fact-verification systems are well explored in the NLP literature with growing attention ow- ing to shared tasks like FEVER. Though the task requires reasoning on extracted evidence to verify a claim’s factuality, there is little work on understanding the reasoning process. In this work, we propose a new methodology for fact-verification, specifically FEVER, that en- forces a closed-world reliance on extracted ev- idence. We present an extensive evaluation of state-of-the-art verification models under these constraints. 1 Introduction A rapid increase in the spread of misinformation on the Internet has necessitated automated solu- tions to determine the validity of a given piece of information. To this end, the Fact Extraction and VERification (FEVER) shared task (Thorne et al., 2018a)1 introduced a dataset for evidence- based fact verification. Given a claim, the task in- volves extracting relevant evidence sentences from a given Wikipedia dump and assigning a label to the claim by reasoning over the extracted evidence (SUPPORTS / REFUTES / NOTENOUGHINFO). Several recent works (Liu et al., 2020; Soleimani et al., 2020; Zhao et al., 2020) leverage representa- tions from large pre-trained language models (LMs) like BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2019) to achieve state-of-the-art results on FEVER. However, it is unclear how factual knowl- edge encompassed in these LMs influences the ver- ification process. More recently, Lee et al. (2020) developed a fact verification system solely based on large pre- trained LMs and presented their superior zero-shot performance on FEVER compared to a random ∗* equal contribution 1https://fever.ai/ baseline. This result clearly shows the influence of factual knowledge embedded inside these LMs, but relying entirely on such knowledge directly contrasts to the evidence-based paradigm of fact- verification. Such reliance can be problematic, es- pecially with evolving evidence (Wikipedia pages are constantly updated to reflect the latest events). Schuster et al. (2019) illustrate this phenomenon through an example fact, “Halep failed to ever win a Wimbledon title”, which was valid until July 2019 but not thereafter. In this work, we propose methods to train fact- verification models that explicitly reason on the available evidence instead of relying on the factual knowledge in pre-trained LMs, thereby emulating a closed-world setting. This is particularly important in the context of the FEVER dataset because of the overlap between the source corpus used for compiling FEVER and the ones commonly used to pre-train LMs (Wikipedia). We build upon the work of Clark et al. (2020) that demonstrated the ability of transformers (BERT, RoBERTa) to function as soft theorem provers. They induce a closed-world reasoning pro- cess by fine-tuning on a carefully curated synthetic natural language rulebase. In this work, we transfer this ability to FEVER and gauge the feasibility of such closed-world reasoning. Additionally, we also construct …

Content truncated to 500 words