MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4685–4697, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4685 MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims Isabelle Augenstein Christina Lioma Dongsheng Wang Lucas Chaves Lima Casper Hansen Christian Hansen Jakob Grue Simonsen Department of Computer Science University of Copenhagen {augenstein,c.lioma,wang,lcl,c.hansen,chrh,simonsen}@di.ku.dk Abstract We contribute the largest publicly available dataset of naturally occurring factual claims for the purpose of automatic claim verification. It is collected from 26 fact checking websites in English, paired with textual sources and rich metadata, and labelled for veracity by hu- man expert journalists. We present an in-depth analysis of the dataset, highlighting character- istics and challenges. Further, we present re- sults for automatic veracity prediction, both with established baselines and with a novel method for joint ranking of evidence pages and predicting veracity that outperforms all base- lines. Significant performance increases are achieved by encoding evidence, and by mod- elling metadata. Our best-performing model achieves a Macro F1 of 49.2%, showing that this is a challenging testbed for claim veracity prediction. 1 Introduction Misinformation and disinformation are two of the most pertinent and difficult challenges of the in- formation age, exacerbated by the popularity of social media. In an effort to counter this, a signif- icant amount of manual labour has been invested in fact checking claims, often collecting the results of these manual checks on fact checking portals or websites such as politifact.com or snopes.com. In a parallel development, researchers have recently started to view fact checking as a task that can be partially automated, using machine learning and NLP to automatically predict the veracity of claims. However, existing efforts either use small datasets consisting of naturally occurring claims (e.g. Mihalcea and Strapparava (2009); Zubiaga et al. (2016)), or datasets consisting of artificially constructed claims such as FEVER (Thorne et al., 2018). While the latter offer valuable contribu- tions to further automatic claim verification work, they cannot replace real-world datasets. Feature Value ClaimID farg-00004 Claim Mexico and Canada assemble cars with foreign parts and send them to the U.S. with no tax. Label distorts Claim URL https://www.factcheck.org/2018/ 10/factchecking-trump-on-trade/ Reason None Category the-factcheck-wire Speaker Donald Trump Checker Eugene Kiely Tags North American Free Trade Agreement Claim Entities United States, Canada, Mexico Article Title FactChecking Trump on Trade Publish Date October 3, 2018 Claim Date Monday, October 1, 2018 Table 1: An example of a claim instance. Entities are obtained via entity linking. Article and outlink texts, evidence search snippets and pages are not shown. Contributions. We introduce the currently largest claim verification dataset of naturally occurring claims.1 It consists of 34,918 claims, collected from 26 fact checking websites in English; evidence pages to verify the claims; the context in which they occurred; and rich metadata (see Table 1 for an example). We perform a thorough analysis to identify characteristics of the dataset such as entities mentioned …

Content truncated to 500 words