Evaluating Unsupervised Representation Learning for Detecting Stances of Fake News

Proceedings of the 28th International Conference on Computational Linguistics, pages 6339–6349 Barcelona, Spain (Online), December 8-13, 2020 6339 Evaluating Unsupervised Representation Learning for Detecting Stances of Fake News Maike Guderlei Department of Statistics Ludwig-Maximilians-Universit¨at Munich, Germany maike@guderlei.de Matthias Aßenmacher Department of Statistics Ludwig-Maximilians-Universit¨at Munich, Germany matthias@stat.uni-muenchen.de Abstract Our goal is to evaluate the usefulness of unsupervised representation learning techniques for de- tecting stances of Fake News. Therefore we examine several pretrained language models with respect to their performance on two Fake News related data sets, both consisting of instances with a headline, an associated news article and the stance of the article towards the respective headline. Specifically, the aim is to understand how much hyperparameter tuning is necessary when finetuning the pretrained architectures, how well transfer learning works in this specific case of stance detection and how sensitive the models are to changes in hyperparameters such as batch size, learning rate (schedule), sequence length as well as the freezing technique. The re- sults indicate that the computationally more expensive autoregression approach of XLNet (Yang et al., 2019) is outperformed by BERT-based models, notably by RoBERTa (Liu et al., 2019). While the learning rate seems to be the most important hyperparameter, experiments with dif- ferent freezing techniques indicate that all evaluated architectures had already learned powerful language representations that pose a good starting point for finetuning them. 1 Introduction With the rise of social media, exchange of opinions and news happens faster than ever. News circulation is therefore less and less bound to traditional print journalism that usually requires extensive research, fact checking and accurate coverage in order to be a reliable news resource. It is relatively easy to share opinions that are either not supported by researched facts or simply wrong. In the worst case a large amount of people can be targeted by propaganda in order to shift societal discussions in favor of a wanted agenda. Human resources are limited to identify such Fake News, since they cover a wide range of topics and linguistic writing styles (Shu et al., 2017, p.2). Automated Fake News detection (FND) has therefore proven to be an important challenge for NLP researchers in recent years. To this day, a variety of approaches dealing with FND exists (Khan et al., 2019). In 2017, the Fake News Challenge Stage 1 was introduced which tackles FND as a stance detection task (Pomerleau and Rao, 2017), where the idea is to determine the stance of a news article to a given headline (Hanselowski et al., 2018, p.1). We now evaluate the five models BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), DistilBERT (Sanh et al., 2019), ALBERT (Lan et al., 2019) and XLNet (Yang et al., 2019) which were developed and enhanced recently. All architectures were pretrained on large unlabeled corpora using a self-supervised objective and can be finetuned on a desired task at hand. Our main focus is to evaluate the necessity of hyperparameter tuning as well as the general performance.1 Besides this, it …

Content truncated to 500 words