STANKER: Stacking Network based on Level-grained Attention-masked BERT for Rumor Detection on Social Media

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3347–3363 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3347 STANKER: Stacking Network based on Level-grained Attention-masked BERT for Rumor Detection on Social Media Dongning Rao1, Xin Miao1, Zhihua Jiang2∗, Ran Li1 1School of Computer, Guangdong University of Technology, Guangzhou 510006, P. R. China 2 Department of Computer Science, Jinan University, Guangzhou 510632, P. R. China raodn@gdut.edu.cn, miaox@mail2.gdut.edu.cn, tjiangzhh@jnu.edu.cn, 2111905160@mail2.gdut.edu.cn Abstract Rumor detection on social media puts pre- trained language models (LMs), such as BERT, and auxiliary features, such as comments, into use. However, on the one hand, rumor detec- tion datasets in Chinese companies with com- ments are rare; on the other hand, intensive interaction of attention on Transformer-based models like BERT may hinder performance improvement. To alleviate these problems, we build a new Chinese microblog dataset named Weibo201 by collecting posts and associated comments from Sina Weibo and propose a new ensemble named STANKER (Stacking neTwork bAsed-on atteNtion-masKed BERT). STANKER adopts two level-grained attention- masked BERT (LGAM-BERT) models as base encoders. Unlike the original BERT, our new LGAM-BERT model takes comments as important auxiliary features and masks co- attention between posts and comments on lower-layers. Experiments on Weibo20 and three existing social media datasets showed that STANKER outperformed all compared models, especially beating the old state-of-the- art on Weibo dataset. 1 Introduction Social media like Sina Weibo is an indispensable part of life, while rumors have severe consequences in political decision-making or manipulating public opinions (Lazer et al., 2018). Therefore, evil-doers can create and spread rumors on social media con- veniently on a massive scale at a low cost (Ma et al., 2020), which provokes a text classification task called rumor detection (Li et al., 2019). For text classification tasks, including rumor detection, transformer-based models like Bidirec- tional Encoder Representations from Transform- ∗Corresponding author. 1Our data and source code are available at: https://github.com/fip-lab/STANKER. All data collec- tion was conducted following the policies of the host institutions’ ethics board. ers (BERT) (Devlin et al., 2018), which achieved impressive results, have a significant performance variation when fine-tuned on small datasets (Risch and Krestel, 2020). Thus researchers proposed ensembles of multiple BERT models (Risch and Krestel, 2020; Fajcik et al., 2019; Liu et al., 2019a) to provide more robust predictions, but a big en- semble size makes the fine-tuning computationally expensive, for the training time and the inference time increase linearly with the ensemble size. Moreover, the attention mechanism, which is a key of Transformer (Vaswani et al., 2017), makes the computational complexity scale quadratically with the input sequence length. It facilitates com- plex models to learn the contextual representation of a word via attending to other words. Encourag- ingly, a few studies indicated that not all attention is necessary (Gordon et al., 2020): partial atten- tion can be pruned (Gordon et al., 2020; Michel et al., 2019) or masked (Liu et al., 2020; Yang et al., 2019) depending on specific tasks, because BERT learns different features at …

Content truncated to 500 words