COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2116–2129 August 1–6, 2021. ©2021 Association for Computational Linguistics 2116 COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic Arkadiy Saakyan1, Tuhin Chakrabarty 1, and Smaranda Muresan1,2 1Department of Computer Science, Columbia University 2Data Science Institute, Columbia University a.saakyan@columbia.edu, {tuhin.chakr,smara}@cs.columbia.edu Abstract We introduce a FEVER-like dataset COVID- Fact of 4, 086 claims concerning the COVID- 19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence. Unlike previ- ous approaches, we automatically detect true claims and their source articles and then gen- erate counter-claims using automatic meth- ods rather than employing human annotators. Along with our constructed resource, we for- mally present the task of identifying relevant evidence for the claims and verifying whether the evidence refutes or supports a given claim. In addition to scientific claims, our data con- tains simplified general claims from media sources, making it better suited for detect- ing general misinformation regarding COVID- 19. Our experiments indicate that COVID-Fact will provide a challenging testbed for the de- velopment of new systems and our approach will reduce the costs of building domain- specific datasets for detecting misinformation. 1 Introduction The proliferation of disinformation and misinfor- mation on the web is increasing at a scale that calls for the automation of the slow and labor- intensive manual fact-checking process (Vosoughi et al., 2018). New York Times reports that “Physi- cians say they regularly treat people more inclined to believe what they read on Facebook than what a medical professional tells them.” Disinformation is even more acute around the recent COVID-19 pandemic. As a result, there is a need for auto- mated fact-checking tools to assist professional fact-checkers and the public in evaluating the ve- racity of claims that are propagated online in news articles or social media. Ideally, a fact-checking pipeline will address sev- eral tasks: 1) Consider real-world claims, 2) Re- trieve relevant documents not bounded to a known Figure 1: A claim from the r/COVID19 subreddit with an academic report as an evidence source linked to it. document collection (e.g., Wikipedia) and which contain information to validate the claim, 3) Se- lect evidence sentences that can support or refute the claim and 4) Predict the claim veracity based on this evidence. Recent work on end-to-end fact- checking, including models and datasets, has ad- vanced the field by addressing several tasks in the pipeline, but not all (Thorne et al., 2018, 2019; Hanselowski et al., 2019; Augenstein et al., 2019; Diggelmann et al., 2021; Wadden et al., 2020). One line of work that includes FEVER (Thorne et al., 2018, 2019) and SciFact (Wadden et al., 2020) addresses tasks 2, 3 and 4, but assumes a given document collection for task 2 (Wikipedia or CORD-19, respectively) and does not address task 1. Moreover, the refuted claims in these datasets are manually generated by asking humans to …

Content truncated to 500 words