TTUI at SemEval-2020 Task 11: Propaganda Detection with Transfer Learning and Ensembles

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1829–1834 Barcelona, Spain (Online), December 12, 2020. 1829 TTUI at SemEval-2020 Task 11: Propaganda Detection with Transfer learning and Ensembles Moonsung Kim and Steven Bethard School of Information College of Social & Behavioral Sciences University of Arizona {moonsungkim,bethard}@arizona.edu Abstract In this paper, we describe our approaches and systems for the SemEval-2020 Task 11 on propaganda technique detection. We fine-tuned BERT and RoBERTa pre-trained models then merged them with an average ensemble. We conducted several experiments for in- put representations dealing with long texts and preserving context as well as for the imbal- anced class problem. Our system ranked 20th out of 36 teams with 0.398 F1 in the SI task and 14th out of 31 teams with 0.556 F1 in the TC task. Our code is available at https://github.com/amenra99/SemEval2020_Task11. 1 Introduction Propaganda is purposeful information aiming to influence an audience to persuade an ideological or political agenda. Propagandistic messages use psychological and rhetorical techniques to hide their intention and can be delivered through materials such as articles, books, images, videos, etc. There have been multiple studies addressing propaganda detection using machine learning systems (Rashkin et al., 2017; Volkova and Jang, 2018; Barr´on-Cedeno et al., 2019; Da San Martino et al., 2019b). More recently, Da San Martino et al. (2019a) organized the NLP4IF-2019 shared task on Fine-Grained Propaganda Detection extended from their previous work, which has a similar framework to this shared task. In SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles, the organizers provided a Propaganda Techniques Corpus (PTC) manually annotated by six professional annotators in 446 news articles from 48 news outlets (Da San Martino et al., 2019b). This shared task is divided into two sub-tasks. The first sub-task is Span Identification (SI) to identify the begin offset and end offset of propaganda fragments in text. This binary sequence tagging task is evaluated by the F1 score of the normalized length of the intersection between manually annotated and predicted spans. The second one is Technique Classification (TC) to label the name of the technique of a given propaganda fragment. In this sub-task, a system is required to classify the specific propaganda fragment into one of the 14 given classes. The original classes were 18 but the organizers merged five underrepresented techniques into two classes to resolve a low proportion of some techniques. This multi-class classification task is evaluated based on the micro-average F1 score. Both sub-tasks share the same dataset and only the last submission on the test set is evaluated per sub-task even though there are no limitations of the number of submissions. A more detailed description of this shared task can be found in Da San Martino et al. (2020). In this paper, we describe our approaches toward dealing with these problems. Two state-of-the-art transformer-based language models, Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) and a Robustly Optimized BERT Pretraining Approach (RoBERTa) (Liu et al., 2019), were used in this shared task. …

Content truncated to 500 words