Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1739–1748 Barcelona, Spain (Online), December 12, 2020. 1739 Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection Gaku Morio∗, Terufumi Morishita*, Hiroaki Ozaki and Toshinori Miyoshi Hitachi, Ltd. Resarch and Development Group Kokubunji, Tokyo, Japan {gaku.morio.vn, terufumi.morishita.wp, hiroaki.ozaki.yu, toshinori.miyoshi.pd}@hitachi.com Abstract In this paper, we show our system for SemEval-2020 task 11, where we tackle propaganda span identification (SI) and technique classification (TC). We investigate heterogeneous pre-trained language models (PLMs) such as BERT, GPT-2, XLNet, XLM, RoBERTa, and XLM-RoBERTa for SI and TC fine-tuning, respectively. In large-scale experiments, we found that each of the lan- guage models has a characteristic property, and using an ensemble model with them is promising. Finally, the ensemble model was ranked 1st amongst 35 teams for SI and 3rd amongst 31 teams for TC. 1 Introduction This paper shows our proposed system for the SemEval-2020 task 11: Detection of Propaganda Tech- niques in News Articles (Da San Martino et al., 2020). The goal of the task was to design a model to detect and classify propaganda. To this end, there are two subtasks: span identification (SI) for predicting propaganda spans and technique classification (TC) for predicting propaganda technique types used for a given span. SI can be assumed as a sequence labeling problem and TC as a multi-label classification problem. Recent studies such as (Yoosuf and Yang, 2019; Vlad et al., 2019) proposed employing BERT (Devlin et al., 2019), a pre-trained language model, for propaganda detection. Since propaganda detection tasks require highly semantic understanding, leveraging such strong pre-trained language models is promis- ing. However, new state-of-the-art pre-trained language models, such as XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), are being proposed rapidly, and there are no sufficient studies on them in the research on propaganda detection. Revealing the ability of capturing propaganda semantics with state-of-the-art models could contribute to discussing future applications. Therefore, we investigate state-of-the-art pre-trained language models (PLMs) for propaganda detec- tion. We employ not only BERT (Devlin et al., 2019) but other PLMs such as GPT-2 (Radford et al., 2019), RoBERTa (Liu et al., 2019), XLM-RoBERTa (Conneau et al., 2019), XLNet (Yang et al., 2019), and XLM (Lample and Conneau, 2019). The PLMs are fine-tuned by our proposed SI and TC models with various hyperparameters. We also propose an effective ensemble method with stacked generaliza- tion, which is generally better than a naive average ensemble. Our ensemble model was ranked 1st in SI and 3rd in TC, showing that leveraging state-of-the-art PLMs is promising. We also empirically gained the following insights as described later. 1.RoBERTa and XLNet generally perform better for propaganda detection. 2.An ensemble model with all types of PLMs showed stable and better performance than employing a single PLM type. 3.Each PLM has a different optimal learning rate, and finding the optimal one is essential to elicit high performance. ∗Contributed equally. This work is licensed under a Creative Commons Attribution 4.0 …

Content truncated to 500 words