FEVER: a Large-scale Dataset for Fact Extraction and VERification

Proceedings of NAACL-HLT 2018, pages 809–819 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics FEVER: a large-scale dataset for Fact Extraction and VERification James Thorne1, Andreas Vlachos1, Christos Christodoulopoulos2, and Arpit Mittal2 1Department of Computer Science, University of Sheffield 2Amazon Research Cambridge {j.thorne, a.vlachos}@sheffield.ac.uk {chrchrs, mitarpit}@amazon.co.uk Abstract In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences ex- tracted from Wikipedia and subsequently verified without knowledge of the sen- tence they were derived from. The claims are classified as SUPPORTED, RE- FUTED or NOTENOUGHINFO by annota- tors achieving 0.6841 in Fleiss κ. For the first two classes, the annotators also recorded the sentence(s) forming the nec- essary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed ora- cles. The best accuracy we achieve on la- beling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we be- lieve that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources. 1 Introduction The ever-increasing amounts of textual informa- tion available combined with the ease in sharing it through the web has increased the demand for ver- ification, also referred to as fact checking. While it has received a lot of attention in the context of journalism, verification is important for other do- mains, e.g. information in scientific publications, product reviews, etc. In this paper we focus on verification of textual claims against textual sources. When compared to textual entailment (TE)/natural language infer- ence (Dagan et al., 2009; Bowman et al., 2015), the key difference is that in these tasks the passage to verify each claim is given, and in recent years it typically consists a single sentence, while in veri- fication systems it is retrieved from a large set of documents in order to form the evidence. Another related task is question answering (QA), for which approaches have recently been extended to han- dle large-scale resources such as Wikipedia (Chen et al., 2017). However, questions typically pro- vide the information needed to identify the answer, while information missing from a claim can of- ten be crucial in retrieving refuting evidence. For example, a claim stating “Fiji’s largest island is Kauai.” can be refuted by retrieving “Kauai is the oldest Hawaiian Island.” as evidence. Progress on the aforementioned tasks has bene- fited from the availability of large-scale datasets (Bowman et al., 2015; Rajpurkar et al., 2016). However, despite the rising interest in verification and fact checking among researchers, the datasets currently used for this task are limited to a few hundred claims. Indicatively, the recently con- ducted Fake News Challenge (Pomerleau and Rao, 2017) with 50 participating teams used a dataset consisting of 300 claims verified against 2,595 as- sociated news articles which is orders of magni- tude …

Content truncated to 500 words