It’s about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits

Findings of the Association for Computational Linguistics: EACL 2023, pages 736–743 May 2-6, 2023 ©2023 Association for Computational Linguistics It’s about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits Yida Mu, Kalina Bontcheva, Nikolaos Aletras Department of Computer Science, The University of Sheffield {y.mu, k.bontcheva, n.aletras}@sheffield.ac.uk Abstract New events emerge over time influencing the topics of rumors in social media. Current ru- mor detection benchmarks use random splits as training, development and test sets which typi- cally results in topical overlaps. Consequently, models trained on random splits may not per- form well on rumor classification on previously unseen topics due to the temporal concept drift. In this paper, we provide a re-evaluation of classification models on four popular rumor de- tection benchmarks considering chronological instead of random splits. Our experimental re- sults show that the use of random splits can sig- nificantly overestimate predictive performance across all datasets and models. Therefore, we suggest that rumor detection models should al- ways be evaluated using chronological splits for minimizing topical overlaps. 1 Introduction Unverified false rumors can spread faster than news from mainstream media, and often can disrupt the democratic process and increase hate speech (Vosoughi et al., 2018; Zubiaga et al., 2018). Au- tomatic detection of rumors is an important task in computational social science, as it helps prevent the spread of false rumors at an early stage (Ma et al., 2017; Zhou et al., 2019; Karmakharm et al., 2019; Bian et al., 2020). Current rumor detection approaches typically rely on existing annotated benchmarks consisting of social media data, e.g., Twitter 15 (Ma et al., 2017), Twitter 16 (Ma et al., 2017), Weibo (Ma et al., 2016), and PHEME (Zubiaga et al., 2016) that cover a wide range of time periods. These benchmarks use random splits for train, develop- ment and test sets which entail some topical overlap among them (see Table 1 for recent previous work). However, the distribution of topics in various NLP benchmarks (e.g., news, reviews, and biomedical) can be significantly affected by time (Huang and Paper Twitter 15 Twitter 16 PHEME Weibo Tian et al. (2022) ✓ ✓ - ✓ Zeng and Gao (2022) - ✓ ✓ - Sheng et al. (2022) - - - ✓ Mukherjee et al. (2022) - - ✓ - Sun et al. (2022) ✓ ✓ ✓ - de Silva and Dou (2021) ✓ ✓ - - Ren et al. (2021) - - ✓ - Wei et al. (2021) ✓ ✓ ✓ - Li et al. (2021) - - ✓ - Rao et al. (2021) ✓ ✓ - ✓ Lin et al. (2021) ✓ ✓ ✓ - Farinneya et al. (2021) - - ✓ - Sun et al. (2021) - - ✓ - Qian et al. (2021) - - ✓ - Song et al. (2021) ✓ ✓ ✓ - Kochkina and Liakata (2020) ✓ ✓ ✓ - Yu et al. (2020) - - ✓ - Xia et al. (2020) - ✓ - ✓ Bian et al. (2020) ✓ ✓ - ✓ …

Content truncated to 500 words