ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1415–1424 Barcelona, Spain (Online), December 12, 2020. 1415 ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them Dawid Jurkiewicz∗ Łukasz Borchmann∗ Izabela Kosmala Filip Grali´nski Applica.ai Zaj˛ecza 15, 00-351 Warsaw, Poland firstname.lastname@applica.ai Abstract This paper presents the winning system for the propaganda Technique Classification (TC) task and the second-placed system for the propaganda Span Identification (SI) task. The purpose of the TC task was to identify an applied propaganda technique given propaganda text fragment. The goal of SI task was to find specific text fragments which contain at least one propaganda technique. Both of the developed solutions used semi-supervised learning technique of self- training. Interestingly, although CRF is barely used with transformer-based language models, the SI task was approached with RoBERTa-CRF architecture. An ensemble of RoBERTa-based models was proposed for the TC task, with one of them making use of Span CLS layers we introduce in the present paper. In addition to describing the submitted systems, an impact of architectural decisions and training schemes is investigated along with remarks regarding training models of the same or better quality with lower computational budget. Finally, the results of error analysis are presented. 1 Introduction The idea of fine-grained propaganda detection was introduced by Da San Martino et al. (2019), whose intention was to facilitate research on this topic by publishing a corpus with detailed annotations of high reliability. There was a chance to propose NLP systems solving this task automatically as a part of this year’s SemEval series. It was expected to detect all fragments of news articles that contain propaganda techniques, and to identify the exact type of used technique (Da San Martino et al., 2020). The authors decided to evaluate Technique Classification (TC) and Span Identification (SI) tasks sepa- rately. The purpose of the TC task was to identify an applied propaganda technique given the propaganda text fragment. In contrast, the goal of the SI task was to find specific text fragments that contain at least one propaganda technique. This paper presents the winning system for the propaganda Technique Clas- sification task and the second-placed system for the propaganda Span Identification task. 2 Systems Description Systems proposed for both SI and TC tasks were based on RoBERTa model (Liu et al., 2019) with task-specific modifications and training schemes applied. The central motif behind our submissions is a commonly used semi-supervised learning technique of self-training (Yarowsky, 1995; Liao and Veeramachaneni, 2009; Liu et al., 2011; Wang et al., 2020), sometimes referred to as incremental semi-supervised training (Rosenberg et al., 2005) or self- learning (Lin et al., 2010). In general, these terms stand for a process of training an initial model on a manually annotated dataset first and using it to further extend the train set by automatically annotating other dataset. Usually, only a selected subset of auto-annotated data is used, however neither selection of high-confidence examples nor loss correction for noisy annotations is performed in our …

Content truncated to 500 words