Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643 June 6–11, 2021. ©2021 Association for Computational Linguistics 624 Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence Tal Schuster Adam Fisch Regina Barzilay Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {tals,fisch,regina}@csail.mit.edu Abstract Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VITAMINC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with addi- tional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VITAMINC are contrastive, i.e., they contain evidence pairs that are nearly identical in lan- guage and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness—improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VITAMINC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evi- dence for verifying the claim, identifying fac- tual revisions, and providing automatic edits via factually consistent text generation.1 1 Introduction Determining the truthfulness of factual claims by comparing them to textual sources of evidence has received intense research interest in recent years. An underlying, but often overlooked, challenge for this paradigm, however, is the dynamic nature of today’s written resources. An extraordinary amount of new information becomes available daily; as a result, many consequential facts are established, changed, or added to over time. We argue that the quality of fact verification systems should be 1The VITAMINC dataset and our models are available at: https://github.com/TalSchuster/VitaminC its population is estimated to be 86,205, almost 14% more than the 2000 census figure of 76,129. its population is estimated to be 91,757, almost 14% more than the 2000 census figure of 76,129. Beaverton, Oregon From Wikipedia, the free encyclopedia Revision ID: 336934876 Revision as of 04:10, 10 January 2010 More than 90K people live in Beaverton Claim: Refutes Supports Figure 1: In VITAMINC, we focus on Wikipedia revi- sions in which the factual content changes. This exam- ple revision now supports an initially refuted claim. measured by how well they adjust to new evidence. In this way, we seek to advance fact verification by requiring that models remain reliable and robust to the change present in practical settings. To this end, we focus on fact verification with contrastive evidence. That is, we infuse the stan- dard fact verification paradigm with challenging cases that require models to be sensitive to fac- tual changes in their presented evidence (hereon referred to interchangeably as “context”). We …

Content truncated to 500 words