Aschern at SemEval-2020 Task 11: It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1462–1468 Barcelona, Spain (Online), December 12, 2020. 1462 aschern at SemEval-2020 Task 11: It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning Anton Chernyavskiy1, Dmitry Ilvovsky1 and Preslav Nakov2 1National Research University Higher School of Economics, Moscow, Russia 2Qatar Computing Research Institute, HBKU, Doha, Qatar aschernyavskiy 1@edu.hse.ru, dilvovsky@hse.ru pnakov@hbku.edu.qa Abstract We describe our system for SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. We developed ensemble models using RoBERTa-based neural architectures, additional CRF layers, transfer learning between the two subtasks, and advanced post-processing to handle the multi-label nature of the task, the consistency between nested spans, repetitions, and labels from similar spans in training. We achieved sizable improvements over baseline fine-tuned RoBERTa models, and the official evaluation ranked our system 3rd (almost tied with the 2nd) out of 36 teams on the span identification subtask with an F1 score of 0.491, and 2nd (almost tied with the 1st) out of 31 teams on the technique classification subtask with an F1 score of 0.62. 1 Introduction The proliferation of disinformation online, commonly known as “fake news”, has given rise to a lot of research on automatic fake news detection. However, most of the efforts have focused on checking whether a piece of information is factually correct, and little attention has been paid to the propaganda techniques that malicious actors use to spread their message. SemEval-2020 Task 11 (Da San Martino et al., 2020a) aims to bridge this gap. It focused on detecting the use of propaganda techniques in news articles,1 creating a dataset that extends (Da San Martino et al., 2019b), and offering two subtasks: • span identification (SI): detecting the propaganda spans in an article; • technique classification (TC): detecting the type of propaganda used in a given text span. Below, we describe the systems we built for these two subtasks. At the core of our systems is RoBERTa (Liu et al., 2019), a pre-trained model based on the Transformer architecture (Vaswani et al., 2017). However, we improved over RoBERTa by adding extra layers in the neural network architecture, and we further added some post-processing steps. We further applied transfer learning between the two subtasks, and finally, we combined different models into an ensemble.2 2 Related Work The dataset for the task comes from (Da San Martino et al., 2019b), which used a BERT-based model with multi-task learning and a gated architecture; the system can be tried online (Da San Martino et al., 2020c). There was also a related previous task on fine-grained propaganda detection (Da San Martino et al., 2019a), where the participants used Transformer-style models, LSTMs and ensembles (Fadel et al., 2019; Hou and Chen, 2019; Hua, 2019). Some approaches further used non-contextualized word embeddings, e.g., based on FastText and GloVe (Gupta et al., 2019; Al-Omari et al., 2019), or handcrafted features such as LIWC, quotes and questions (Alhindi et al., 2019). For the fragment-classification subtask (a combination of two subtasks of the current …

Content truncated to 500 words