TakeLab at SemEval-2019 Task 4: Hyperpartisan News Detection

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 995–998 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 995 TakeLab at SemEval-2019 Task 4: Hyperpartisan News Detection Niko Pali´c, Juraj Vladika, Dominik ˇCubeli´c, Ivan Lovrenˇci´c, Maja Buljan, Jan ˇSnajder Text Analysis and Knowledge Engineering Lab Faculty of Electrical Engineering and Computing, University of Zagreb Unska 3, 10000 Zagreb, Croatia {name.surname}@fer.hr borat-sagdiyev team Abstract In this paper, we demonstrate the system built to solve the SemEval-2019 task 4: Hyperpar- tisan News Detection (Kiesel et al., 2019), the task of automatically determining whether an article is heavily biased towards one side of the political spectrum. Our system receives an ar- ticle in its raw, textual form, analyzes it, and predicts with moderate accuracy whether the article is hyperpartisan. The learning model used was primarily trained on a manually pre- labeled dataset containing news articles. The system relies on the previously constructed SVM model, available in the Python Scikit- Learn library. We ranked 6th in the compe- tition of 42 teams with an accuracy of 79.1% (the winning team had 82.2%). 1 Introduction The ability to quickly, precisely, and efficiently discern if a given article is hyperpartisan can prove to be beneficial in a multitude of different scenar- ios. Should we, for example, wish to evaluate if a certain news publisher delivers politically biased content, the best way to do so would be analyz- ing that very content. However, the sheer amount of articles modern news companies produce nowa- days asks for an automated approach to the prob- lem. Spotting bias in text is both a well-known and challenging natural language problem. As bias can manifest itself in a covert or ambiguous manner, it is often hard even for an experienced reader to detect it. There was some research done on similar issues before (Doumit and Minai, 2011), but none specifically on the subject of hyperpartisan news. The system described in this paper was built for Task 4 of the SemEval-2019 competition. The goal of the system, as set by the task, is to pre- dict, as accurately as possible, whether a given ar- ticle is hyperpartisan. While there were other cri- teria for evaluating the performance of the model (precision, recall, F1), we decided to optimize the program for the accuracy criterion, as the rankings were based solely on this measure. Accuracy, in this context, indicates the ratio of correctly pre- dicted articles to the total number of articles. The final model reached an accuracy of 79.1%, which presents a decent score, considering the complex- ity of the problem and the available technology. The system we built is based on the SVM model publicly available in Python’s SciKit-Learn library (Pedregosa et al., 2011). Our model was trained on a handful of carefully chosen features derived from the given dataset and our understanding of the nature of bias. The dataset was split into a high-quality, manually labelled set of articles, and a large, but sub-par set of automatically …

Content truncated to 500 words