A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 493–503 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 493 A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking Andreas Hanselowski†∗, Christian Stab†∗, Claudia Schulz†∗, Zile Li∗, Iryna Gurevych†∗ †Research Training Group AIPHES https://www.aiphes.tu-darmstadt.de ∗Ubiquitous Knowledge Processing Lab (UKP-TUDA) https://www.ukp.tu-darmstadt.de/ †∗ Computer Science Department, Technische Universit¨at Darmstadt Abstract Automated fact-checking based on machine learning is a promising approach to identify false information distributed on the web. In order to achieve satisfactory performance, ma- chine learning methods require a large corpus with reliable annotations for the different tasks in the fact-checking process. Having analyzed existing fact-checking corpora, we found that none of them meets these criteria in full. They are either too small in size, do not provide de- tailed annotations, or are limited to a single domain. Motivated by this gap, we present a new substantially sized mixed-domain corpus with annotations of good quality for the core fact-checking tasks: document retrieval, evi- dence extraction, stance detection, and claim validation. To aid future corpus construc- tion, we describe our methodology for corpus creation and annotation, and demonstrate that it results in substantial inter-annotator agree- ment. As baselines for future research, we per- form experiments on our corpus with a number of model architectures that reach high perfor- mance in similar problem settings. Finally, to support the development of future models, we provide a detailed error analysis for each of the tasks. Our results show that the realistic, multi-domain setting defined by our data poses new challenges for the existing models, pro- viding opportunities for considerable improve- ment by future systems. 1 Introduction The ever-increasing role of the Internet as a pri- mary communication channel is arguably the sin- gle most important development in the media over the past decades. While it has led to unprece- dented growth in information coverage and distri- bution speed, it comes at a cost. False informa- tion can be shared through this channel reaching a much wider audience than traditional means of disinformation (Howell et al., 2013). While human fact-checking still remains the primary method to counter this issue, the amount and the speed at which new information is spread makes manual validation challenging and costly. This motivates the development of automated fact- checking pipelines (Thorne et al., 2018a; Popat et al., 2017; Hanselowski and Gurevych, 2017) consisting of several consecutive tasks. The fol- lowing four tasks are commonly included in the pipeline. Given a controversial claim, document retrieval is applied to identify documents that con- tain important information for the validation of the claim. Evidence extraction aims at retrieving text snippets or sentences from the identified doc- uments that are related to the claim. This evidence can be further processed via stance detection to infer whether it supports or refutes the claim. Fi- nally, claim validation assesses the validity of the claim given the evidence. Automated fact-checking has received signifi- cant attention in the NLP community in the past years. …

Content truncated to 500 words