Explainable Automated Fact-Checking for Public Health Claims

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7740–7754, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7740 Explainable Automated Fact-Checking for Public Health Claims Neema Kotonya and Francesca Toni Department of Computing Imperial College London, United Kingdom {nk2418,ft}@ic.ac.uk Abstract Fact-checking is the task of verifying the ve- racity of claims by assessing their assertions against credible evidence. The vast major- ity of fact-checking studies focus exclusively on political claims. Very little research ex- plores fact-checking for other topics, specif- ically subject matters for which expertise is required. We present the first study of ex- plainable fact-checking for claims which re- quire specific expertise. For our case study we choose the setting of public health. To sup- port this case study we construct a new dataset PUBHEALTH of 11.8K claims accompanied by journalist crafted, gold standard explanations (i.e., judgments) to support the fact-check la- bels for claims1. We explore two tasks: ve- racity prediction and explanation generation. We also define and evaluate, with humans and computationally, three coherence properties of explanation quality. Our results indicate that, by training on in-domain data, gains can be made in explainable, automated fact-checking for claims which require specific expertise. 1 Introduction A great amount of progress has been made in the area of automated fact-checking. This in- cludes more accurate machine learning models for veracity prediction and datasets of both nat- urally occurring (Wang, 2017; Augenstein et al., 2019; Hanselowski et al., 2019) and human-crafted (Thorne et al., 2018) fact-checking claims, against which the models can be evaluated. However, a few blind spots exist in the state-of-the-art. In this work we address specifically two shortcomings: the nar- row focus on political claims, and the paucity of explainable systems. One subject area which we believe could benefit from expertise-based fact-checking is public health 1Data and code are available here: https://github. com/neemakot/Health-Fact-Checking – including the study of epidemiology, disease pre- vention in a population, and the formulation of public policies (Turnock, 2012). Recent events, including the COVID-19 pandemic, demonstrate the significant potential harm of misinformation in the public health setting, and the importance in accurately fact-checking claims. Unlike political and general misinformation, specific expertise is re- quired in order to fact check claims in this domain. Oftentimes this expertise may be limited, and thus claims which surround public health may be in- accessible (e.g., because of the use of jargon and biomedical terminology) in a way political claims are not. Nonetheless, like political misinformation, the public health variety is also potentially very dangerous, because it can put people in imminent danger and risk lives. Typically, statements which are candidates for fact-checking originate in the political domain (Vla- chos and Riedel, 2014; Ferreira and Vlachos, 2016; Wang, 2017), and tend to surround more general topics or be non-subject specific (Thorne et al., 2018). This follows the trend of the rising interest in political fact-checking in the last decade (Graves, 2018). There are on-going efforts with respect to fact-checking scientific claims (Grabitz et …

Content truncated to 500 words