CyberWallE at SemEval-2020 Task 11: An Analysis of Feature Engineering for Ensemble Models for Propaganda Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1469–1480 Barcelona, Spain (Online), December 12, 2020. 1469 CyberWallE at SemEval-2020 Task 11: An Analysis of Feature Engineering for Ensemble Models for Propaganda Detection Verena Blaschke Maxim Korniyenko Seminar f¨ur SprachwissenschaftSam Tureski Eberhard Karls Universit¨at T¨ubingen first.last@student.uni-tuebingen.de Abstract This paper describes our participation in the SemEval-2020 task Detection of Propaganda Tech- niques in News Articles. We participate in both subtasks: Span Identification (SI) and Technique Classification (TC). We use a bi-LSTM architecture in the SI subtask and train a complex en- semble model for the TC subtask. Our architectures are built using embeddings from BERT in combination with additional lexical features and extensive label post-processing. Our systems achieve a rank of 8 out of 35 teams in the SI subtask (F1-score: 43.86%) and 8 out of 31 teams in the TC subtask (F1-score: 57.37%). 1 Introduction Propaganda is defined as “the deliberate and systematic attempt to shape perceptions, manipulate cogni- tions, and direct behavior to achieve a response that furthers the desired intent of the propagandist” (Jowett and O’Donnell, 2019, p. 6). With the advent of rapid dissemination of news articles through online social media, automatic detection of biased or fake reporting has become more crucial than ever before. This paper describes our participation in both of the subtasks offered by Da San Martino et al. (2020) in the SemEval 2020 shared task for the Detection of Propaganda Techniques in News Articles. The Span Identification (SI) subtask is a binary classification problem to discover propaganda at the token level, and the Technique Classification (TC) subtask involves a 14-way classification of propagandistic text fragments. To address the SI task, we combine token-level BERT embeddings and linguistic features with bidi- rectional LSTMs and data post-processing methods. To address the TC task, we use BERT sequence embeddings and linguistic features to train a feed-forward neural network before post-processing is applied. While top-scoring teams in a similar shared task (Da San Martino et al., 2019b) focus primarily on leveraging the performance of the pre-trained, context-dependent language model BERT, we find that further encoding of linguistic features offers a meaningful boost over both BERT and GloVe word embeddings alone. Majority voting and span merging in the SI task, as well as pre- and post-processing techniques to account for frequent occurrences of the REPETITION technique in the TC task result in further performance increases. Along with our best-performing model, we provide an extensive exploration into various embedding, feature and classifier combinations. We release our code at github.com/cicl-iscl/CyberWallE-propaganda-detection. 2 Background Before the introduction of the 2019 shared task on propaganda detection, approaches to propaganda recognition in news generally focused on classification at the article level. Rashkin et al. (2017) compare the language of “trusted news articles” from a well-known corpus with stories from “unreliable news sites”. The publicly-available, feature-based tool Proppy, released by Barr´on-Cedeno et al. (2019), ranks articles by their likelihood of containing propagandist content. These previous systems have been prone This work is licensed under …

Content truncated to 500 words