Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification Using Pre-trained Language Models

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1448–1455 Barcelona, Spain (Online), December 12, 2020. 1448 Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models Shuohuan Wang, Jiaxiang Liu, Xuan Ouyang, Yu Sun Baidu Inc., China {wangshuohuan,liujiaxiang,ouyangxuan,sunyu02}@baidu.com Abstract This paper describes Galileo’s performance in SemEval-2020 Task 12 on detecting and catego- rizing offensive language in social media. For Offensive Language Identification, we proposed a multi-lingual method using Pre-trained Language Models, ERNIE and XLM-R. For offensive language categorization, we proposed a knowledge distillation method trained on soft labels gen- erated by several supervised models. Our team participated in all three sub-tasks. In Sub-task A - Offensive Language Identification, we ranked first in terms of average F1 scores in all languages. We are also the only team which ranked among the top three across all languages. We also took the first place in Sub-task B - Automatic Categorization of Offense Types and Sub-task C - Offence Target Identification. 1 Introduction Due to the growing number of Internet users, cyber-violence emerged with offensive language pervasive across social media. With anonymity as a “privilege”, netizens hide behind the screens, behaving in a manner most of them would not otherwise in reality. Thus, government organizations, online communities, and technology companies are all striving for ways to detect aggressive language in social media and help build a more friendly online environment. Manual filtering is very time consuming and it can cause post-traumatic stress disorder-like symptoms to human annotators. One of the most common strategies (Waseem et al., 2017; Davidson et al., 2017; Malmasi and Zampieri, 2018; Kumar et al., 2018) to tackle the problem is to train systems capable of recognizing offensive content, which can then be deleted or set aside human moderation. SemEval 2020 Task-12 (Zampieri et al., 2020) is the second edition of OffensEval (Zampieri et al., 2019). In this competition, organizers offers 5 languages datasets including Arabic (Mubarak et al., 2020), Danish (Sigurbergsson and Derczynski, 2020), English (Rosenthal et al., 2020), Turkish (C¸ ¨oltekin, 2020) and Greek (Pitenis et al., 2020). In Sub-task A, the participants need to predict whether a post uses offensives language. Besides, the organizers provide other two sub-tasks which mainly focus on English, to predict the type and target of offensive language. Participating in all 3 Sub-tasks, we proposed several methods based on pre-training language models including ERNIE and XLM-R. In Sub-task A, we scored 0.9199, 0.851, 0.8258, 0.802, 0.8989 in English, Greek, Turkish, Danish and Arabic respectively. We ranked first in average F1 scores, and ranked in top three across all languages. In Sub-task B and Sub-task C, we also took the first place with 0.7462 and 0.7145. In the following sections, we will elaborate the methods, dataset and experiments of our system. 2 Relate Work 2.1 Monolingual Pre-trained Language Models Pre-training and fine-tuning have become a new paradigm in natural language processing, where the general knowledge is firstly learnt from large-scale corpus through self-supervised learning and then This work …

Content truncated to 500 words