Dick-Preston and Morbo at SemEval-2019 Task 4: Transfer Learning for Hyperpartisan News Detection

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 939–943 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 939 Dick-Preston and Morbo at SemEval-2019 Task 4: Transfer Learning for Hyperpartisan News Detection Tim Isbister Swedish Defence Research Agency Stockholm, Sweden tim.isbister@foi.se Fredrik Johansson Swedish Defence Research Agency Stockholm, Sweden fredrik.johansson@foi.se Abstract In a world of information operations, influence campaigns, and fake news, classification of news articles as following hyperpartisan argu- mentation or not is becoming increasingly im- portant. We present a deep learning-based ap- proach in which a pre-trained language model has been fine-tuned on domain-specific data and used for classification of news articles, as part of the SemEval-2019 task on hyper- partisan news detection. The suggested ap- proach yields accuracy and F1-scores around 0.8 which places the best performing classifier among the top-5 systems in the competition. 1 Introduction In today’s polarized media and political land- scapes, the challenge of determining whether a news article is biased or not is highly topical. In the hyperpartisan news detection task (Kiesel et al., 2019) of the International Workshop on Se- mantic Evaluation (SemEval) 2019, the task is to predict whether a given news article text follows a hyperpartisan (extreme one-sided) argumentation or not, i.e., whether it exhibits blind or prejudiced allegiance to one party, cause, or person (Potthast et al., 2019). As part of this challenge, participat- ing research teams got access to two datasets: 1. by-publisher: A well-balanced dataset con- sisting of 750,000 articles in which the data have been labeled by the overall bias of the publisher, as provided by journalists or fact- checking sites. 2. by-article: A smaller dataset consisting of 645 articles for which crowdsourcing work- ers have agreed on the labeling of the articles as being hyperpartisan (37%) or not (63%). A similar but more well-balanced test dataset (to which the participating teams have not got direct access) has been used for evaluating the accuracy, precision, recall, and F1-score of systems developed by the participating re- search teams. In this system description paper we present the results for the two participating research teams from the Swedish Defence Research Agency (FOI): 1) dick-preston and 2) morbo. The teams contributed with separate systems for the early-bird deadline and for the final submis- sion. In the early phase we used traditional ma- chine learning classifiers such as logistic regres- sion and support vector machines (SVMs), built upon traditional text features such as word and character n-gram term frequencies (weighted with inverse document frequency). These classifiers have been used as baselines to which more “mod- ern” NLP classifiers have been compared. For the final submission both teams made use of transfer learning-based Universal Language Model Fine- Tuning (ULMFiT) models. The difference in the teams’ final systems is the percentage of data used for training/validation splits when fine-tuning the models and the number of epochs for which the models were trained. Despite that only a few hundred examples were used for fine-tuning the pre-trained ULMFiT-models, accuracies …

Content truncated to 500 words