Quevedo: Annotation and Processing of Graphical Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 244–257 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 244 CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweets Isabelle Mohr, Amelie W¨uhrl, Roman Klinger Institut f¨ur Maschinelle Sprachverarbeitung, University of Stuttgart, Germany {isabelle.mohr, amelie.wuehrl, roman.klinger}@ims.uni-stuttgart.de Abstract Over the course of the COVID-19 pandemic, large volumes of biomedical information concerning this new disease have been published on social media. Some of this information can pose a real danger to people’s health, particularly when false information is shared, for instance recommendations on how to treat diseases without professional medical advice. Therefore, automatic fact-checking resources and systems developed specifically for the medical domain are crucial. While existing fact-checking resources cover COVID-19-related information in news or quantify the amount of misinformation in tweets, there is no dataset providing fact-checked COVID-19-related Twitter posts with detailed annotations for biomedical entities, relations and relevant evidence. We contribute CoVERT, a fact-checked corpus of tweets with a focus on the domain of biomedicine and COVID-19-related (mis)information. The corpus consists of 300 tweets, each annotated with medical named entities and relations. We employ a novel crowdsourcing methodology to annotate all tweets with fact-checking labels and supporting evidence, which crowdworkers search for online. This methodology results in moderate inter-annotator agreement. Furthermore, we use the retrieved evidence extracts as part of a fact-checking pipeline, finding that the real-world evidence is more useful than the knowledge indirectly available in pretrained language models. Keywords: fact-checking, bioNLP, social media mining, COVID-19 1. Introduction The COVID-19 pandemic has elicited a global response in the scientific community, leading to a burst of new information regarding the pathophysiology of the virus, new treatments for infected patients as well as vaccines in production (Chahrour et al., 2020). At the same time, large volumes of information about COVID-19 have also been published on social media platforms (Yang et al., 2020; Kouzy et al., 2020; Singh et al., 2020; Shahi et al., 2021), which are known to reach large networks of individuals in short amounts of time (Kouzy et al., 2020). This reach is paramount to curbing the spread of a virus like SARS- CoV-2. Although much of the information circulating online is potentially useful, some of it, in particular misinformation and disinformation, can also pose a great danger (Soltaninejad, 2020). Incorrect health advice can not only put individuals at risk, false and misleading information may also be detrimental to efforts in controlling the pandemic. Therefore, the stream of information on social media sites requires critical thought and fact-checking. Manual fact-checking, however, is time-consuming and expensive. One way to perform the task automatically is to use machine learning to classify a claim to be true or false. An intermediate step to exploit potential evidence is to automatically decide for claim-evidence pairs if the evidence supports or refutes the claim (or whether that information is insufficient to rule a verdict) (Vlachos and Riedel, 2014). The growing need for fact-checking of COVID- 19-related information has …

Content truncated to 500 words