UMSIForeseer at SemEval-2020 Task 11: Propaganda Detection by Fine-Tuning BERT with Resampling and Ensemble Learning

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1841–1846 Barcelona, Spain (Online), December 12, 2020. 1841 UMSIForeseer at SemEval-2020 Task 11: Propaganda Detection by Fine-Tuning BERT with Resampling and Ensemble Learning Yunzhe Jiang1, Cristina Gˆarbacea2, Qiaozhu Mei1,2 1School of Information, University of Michigan, Ann Arbor, MI, USA 2Department of EECS, University of Michigan, Ann Arbor, MI, USA {yunzhej, garbacea, qmei}@umich.edu Abstract We describe our participation at the SemEval 2020 “Detection of Propaganda Techniques in News Articles” - Techniques Classification (TC) task, designed to categorize textual fragments into one of the 14 given propaganda techniques. Our solution leverages pre-trained BERT models. We present our model implementations, evaluation results and analysis of these results. We also investigate the potential of combining language models with resampling and ensemble learning methods to deal with data imbalance and improve performance. 1 Introduction Propaganda techniques are used to promote or publicize a particular political cause or point of view, especially of a biased or misleading nature (Orlov and Litvak, 2018). To achieve its desired outcome, psychological and rhetorical techniques are frequently used. While initially people tend to agree with propaganda messages due to their misuse of logic and/ or arousal of emotional response, they later change their opinion and realize arguments are not convincing. Indeed, for maximum effect propaganda techniques are intended to go unnoticed. It is therefore important to detect propaganda in initial stages and identify the specific propaganda techniques used. By successfully detecting and classifying propaganda, people can look at information more rationally and logically. This paper describes the solutions towards the Techniques Classification (TC) task of the SemEval 2020 Task 11 “Detection of Propaganda Techniques in News Articles” competition. The task requires classifying textual fragments that relate to at least one of the 14 given propaganda techniques. Our solutions leverage the pre-trained BERT (Devlin et al., 2018) based classifier. Moreover, fine-tuning allows us to conveniently adjust its pre-trained bottom layer weights on the given propaganda detection corpus shared by the organizers. Among the 46 participating teams, our submission is ranked 17. The rest of the paper is organized as follows. In Section 2 we give the problem definition. In Section we present the propaganda dataset, in Section 4 we describe our solution and elaborate on the architectural and implementation details of the model used, in Section 5 we present results achieved by our best submission and analyze of these results, and finally in Section 6 we conclude with directions for future work. 2 Problem Definition Given a document and a propaganda-related text excerpt from the document, the task is to identify the specific propaganda technique present in the text excerpt from the 14 propaganda classes available (Da San Martino et al., 2020). Text excerpts can be overlapping and reflect multiple propaganda techniques simultaneously, therefore the propaganda identification task needs to be approached as a multi-label multi- class classification problem. In practice, for cases when multiple propaganda techniques are used in the same text fragment, the organizers created as many copies of …

Content truncated to 500 words