SkoltechNLP at SemEval-2020 Task 11: Exploring Unsupervised Text Augmentation for Propaganda Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1786–1792 Barcelona, Spain (Online), December 12, 2020. 1786 SkoltechNLP at SemEval-2020 Task 11: Exploring Unsupervised Text Augmentation for Propaganda Detection Daryna Dementieva, Igor Markov, and Alexander Panchenko Skolkovo Institute of Science and Technology Bol’shoy Bul’var, 30, Skolkovo Innovation Center, Moscow Oblast, 143026, Russia {Daryna.Dementieva, Igor.Markov, A.Panchehko}@skoltech.ru Abstract This paper presents a solution for the Span Identification (SI) task in the “Detection of Propaganda Techniques in News Articles” competition at SemEval-2020. The goal of the SI task is to identify specific fragments of each article which contain the use of at least one propaganda technique. This is a binary sequence tagging task. We tested several approaches finally selecting a fine-tuned BERT model as our baseline model. Our main contribution is an investigation of several unsupervised data augmentation techniques based on distributional semantics expanding the original small training dataset as applied to this BERT-based sequence tagger. We explore various expansion strategies and show that they can substantially shift the balance between precision and recall, while maintaining comparable levels of the F1 score. 1 Introduction Propaganda is one of the primary indicators of false news. Therefore, the task of detecting and classifying propaganda is an important one in the field of fake news detection. Da San Martino et al. (2019) presented a new dataset for propaganda detection. The main advantage of this dataset is the markup at the level of individual text fragments. In addition to binary markup (is/is not propaganda), there is also a multiclass markup, which includes 18 different propaganda classes. Using this dataset, the Span Identification (SI) task of Detection of Propaganda Techniques in News Articles task requires to determine specific text fragments which contain at least one propaganda technique. For instance, in the sentence “Manchin says Democrats acted like babies at the SOTU (video) Personal Liberty PollExercise your right to vote” the part from the 34-th through the 40-th character (i.e., word “babies”) belongs to the Name-calling and Labeling class, so it should be marked as propaganda. This paper describes the solution by “SkoltechNLP” team which took part in the SI task of the competition and achieved a score F1 = 0.34 on the test set evaluation. Our solution is based on BERT (Devlin et al., 2019) that is specially pretrained for the sequence tagging task. However, such architectures usually require large datasets for fine-tuning: for instance, CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003) consists of 976 articles in the training set, in comparison with 371 articles in the training set for the propaganda detection task. Therefore, we conducted a study of the dataset expansion with several augmentation techniques. We built a number of strategies, performing investigation of various combinations of such parameters as: the size of the enlarged dataset, models for word representations, words’ part of speech, and class for substitution. All codes for reproducing the results are openly available online.1 The rest of the paper is organized as follows. Section 2 covers several previous approaches to …

Content truncated to 500 words