Fact or Fiction: Verifying Scientific Claims

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7534–7550, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7534 Fact or Fiction: Verifying Scientific Claims David Wadden†∗ Shanchuan Lin† Kyle Lo‡ Lucy Lu Wang‡ Madeleine van Zuylen‡ Arman Cohan‡ Hannaneh Hajishirzi†‡ † University of Washington, Seattle, WA, USA ‡ Allen Institute for Artificial Intelligence, Seattle, WA, USA {dwadden,linsh,hannaneh}@cs.washington.edu {kylel,lucyw,madeleinev,armanc}@allenai.org Abstract We introduce scientific claim verification, a new task to select abstracts from the re- search literature containing evidence that SUP- PORTS or REFUTES a given scientific claim, and to identify rationales justifying each de- cision. To study this task, we construct SCI- FACT, a dataset of 1.4K expert-written scien- tific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SCIFACT, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 cor- pus. Our experiments indicate that SCIFACT will provide a challenging testbed for the de- velopment of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https:// github.com/allenai/scifact. A leader- board and COVID-19 fact-checking demo are available at https://scifact.apps. allenai.org. 1 Introduction Due to rapid growth in the scientific literature, it is difficult for researchers – and the general pub- lic even more so – to stay up to date on the latest findings. This challenge is especially acute during public health crises like the current COVID-19 pan- demic, due to the extremely fast rate at which new findings are reported and the risks associated with making decisions based on outdated or incomplete information. As a result, there is a need for auto- mated tools to assist researchers and the public in evaluating the veracity of scientific claims. ∗Work performed during internship with the Allen Insti- tute for Artificial Intelligence. More severe COVID-19 infection is associated with higher mean troponin (SMD 0.53, 95% CI 0.30 to 0.75, p < 0.001) Decision: SUPPORTS Claim Fact-checker Rationale Corpus Cardiac injury is common in critical cases of COVID-19. Figure 1: A scientific claim, supported by evidence identified by our system. To correctly verify this claim, the system must possess background knowledge that troponin is a protein found in cardiac muscle and that elevated levels of troponin are a marker of cardiac injury. In addition, it must be able to reason about di- rectional relationships between scientific processes: re- placing higher with lower would cause the rationale to REFUTE the claim rather than SUPPORT it. Finally, the system should interpret p < 0.001 as an indication that the reported finding is statistically significant. Fact-checking – a task in which the veracity of an input claim is verified against a corpus of documents that support or refute the claim – has been studied to combat the proliferation of misin- formation in political …

Content truncated to 500 words