FACTOID: A New Dataset for Identifying Misinformation Spreaders and Political Bias

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3231–3241 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3231 FACTOID: A New Dataset for Identifying Misinformation Spreaders and Political Bias Flora Sakketou∗†, Joan Plepi∗†, Riccardo Cervero‡, Henri-Jacques Geiss ⋄, Paolo Rosso‡ , Lucie Flek† †Conversational AI and Social Analytics (CAISA) Lab Department of Mathematics and Computer Science, University of Marburg, Germany ‡ Pattern Recognition and Human Language Technology (PRHLT) Research Center Universitat Polit`ecnica de Val`encia, Spain ⋄ Department of Computer Science, Technical University of Darmstadt ∗ These authors contributed equally to this work {flora.sakketou, joan.plepi, lucie.flek}@uni-marburg.de, {rcerver1@upvnet, prosso@dsic}.upv.es henri-jacques.geiss@stud.tu-darmstadt.de Abstract Proactively identifying misinformation spreaders is an important step towards mitigating the impact of fake news on our society. In this paper, we introduce a new contemporary Reddit dataset for fake news spreader analysis, called FACTOID, monitoring political discussions on Reddit since the beginning of 2020. The dataset contains over 4K users with 3.4M Reddit posts, and includes, beyond the users’ binary labels, also their fine-grained credibility level (very low to very high) and their political bias strength (extreme right to extreme left). As far as we are aware, this is the first fake news spreader dataset that simultaneously captures both the long-term context of users’ historical posts and the interactions between them. To create the first benchmark on our data, we provide methods for identifying misinformation spreaders by utilizing the social connections between the users along with their psycho-linguistic features. We show that the users’ social interactions can, on their own, indicate misinformation spreading, while the psycho-linguistic features are mostly informative in non-neural classification settings. In a qualitative analysis we observe that detecting affective mental processes correlates negatively with right-biased users, and that the openness to experience factor is lower for those who spread fake news. Keywords: fake news spreader detection, fake news and political bias dataset, Reddit 1. Introduction As the popularity of social media platforms continu- ously grows, so does the dissemination of online dis- information. Many deep learning systems have been therefore developed to detect false or biased news (Zhou and Zafarani, 2020; Zellers et al., 2019; Monti et al., 2019; Shu et al., 2017). While fake news detection is a big step to mitigate the impact of misinformation on our society (Figueira and Oliveira, 2017; Visentin et al., 2019), it is not sufficient, since limiting the diffusion of false information and avoiding its catastrophic effects is extremely challenging, especially once it has been shared on the Web (Cheng and Chen, 2020; McKay and Tenove, 2020). Research shows that fact corrections frequently fail in reducing people’s misconception of the truth, and occasionally they even have a “backfir- ing” effect where people’s misconception is reinforced (Redlawsk et al., 2010; Nyhan and Reifler, 2010; Swire et al., 2017; Berinsky, 2017). It is essential to address this issue at its origin - to ef- ficiently and rapidly identify accounts and users which are likely to propagate posts from the handles of …

Content truncated to 500 words