Telling a Lie: Analyzing the Language of Information and Misinformation during Global Health Events

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4135–4141 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4135 Telling a Lie: Analyzing the Language of Information and Misinformation during Global Health Events Ankit Aich and Natalie Parde Natural Language Processing Laboratory Department of Computer Science University of Illinois at Chicago {aaich2, parde}@uic.edu Abstract The COVID-19 pandemic and other global health events are unfortunately excellent environments for the creation and spread of misinformation, and the language associated with health misinformation may be typified by unique patterns and linguistic markers. Allowing health misinformation to spread unchecked can have devastating ripple effects; however, detecting and stopping its spread requires careful analysis of these linguistic characteristics at scale. We analyze prior investigations focusing on health misinformation, associated datasets, and detection of misinformation during health crises. We also introduce a novel dataset designed for analyzing such phenomena, comprised of 2.8 million news articles and social media posts spanning the early 1900s to the present. Our annotation guidelines result in strong agreement between independent annotators. We describe our methods for collecting this data and follow this with a thorough analysis of the themes and linguistic features that appear in information versus misinformation. Finally, we demonstrate a proof-of-concept misinformation detection task to establish dataset validity, achieving a strong performance benchmark (accuracy = 75%; F1 = 0.7). Keywords: Dataset, Misinformation, Disinformation 1. Introduction Just like viruses themselves, misinformation from small clusters of individuals can quickly evolve to global, com- munity spread. Information may propagate rapidly via word of mouth, social media, and news articles, cre- ating confusion and dangerous situations. This phe- nomenon has been recognized as an infodemic by the World Health Organization.1 Despite the urgency of developing automated methods capable of identifying and extinguishing misinformation before it spreads too widely, few resources exist to facilitate the development of such models, especially as they pertain to generaliz- able settings beyond the COVID-19 pandemic. In this work, we introduce a new, large dataset to stimu- late such research. We collect 2.8 million news articles and (when available) social media posts covering the following diseases: small pox, the 1918 Spanish flu, MERS, SARS, H1N1, Ebola, HIV (AIDS), and COVID- 19. News articles from traditional print media range from the early 1900s to the present, and are sourced from long-standing print houses including The New York Times,2 the British Broadcasting Corporation,3 Cable News Network (CNN),4 The Washington Post,5 and Al Jazeera,6 as well as the Centers for Disease Control and Prevention (CDC)7 and the World Health Organization 1https://www.who.int/news-room/spotli ght/let-s-flatten-the-infodemic-curve 2https://www.nytimes.com 3https://www.bbc.com 4https://www.cnn.com 5https://www.washingtonpost.com 6https://www.aljazeera.com 7https://www.cdc.gov (WHO). Social media coverage of more recent diseases is sourced from Twitter.8 Our primary contributions are as follows: • We analyze prior work investigating health infor- mation (with a focus on the ongoing COVID-19 pandemic), existing misinformation datasets, and automated detection of misinformation, identifying gaps and important future directions (§2). • We introduce a new, large (2.8 million instances) collection of news articles and social …

Content truncated to 500 words