FBK-DH at SemEval-2020 Task 12: Using Multi-channel BERT for Multilingual Offensive Language Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1539–1545 Barcelona, Spain (Online), December 12, 2020. 1539 FBK-DH at SemEval-2020 Task 12: Using Multi-channel BERT for Multilingual Offensive Language Detection Camilla Casula†, Alessio Palmero Aprosio‡, Stefano Menini‡, Sara Tonelli‡ †Dept. of Linguistics and Philology, Uppsala University, Sweden ‡Fondazione Bruno Kessler (FBK), Trento, Italy camillacasula@gmail.com {aprosio,menini,satonelli}@fbk.eu Abstract In this paper we present our submission to subtask A at SemEval 2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval2). For Danish, Turkish, Arabic and Greek, we develop an architecture based on transfer learning and relying on a two-channel BERT model, in which the English BERT and the multilingual one are combined after creating a machine-translated parallel corpus for each language in the task. For English, instead, we adopt a more standard, single-channel approach. We find that, in a multilingual scenario, with some languages having small training data, using parallel BERT models with machine translated data can give systems more stability, especially when dealing with noisy data. The fact that machine translation on social media data may not be perfect does not hurt the overall classification performance. 1 Introduction Concealed by perceived anonymity, users can often feel comfortable expressing offensive, abusive and hateful thoughts on the Internet. Therefore, the task of identifying offensive language on social media has increasingly gained attention in recent years, since the manual identification and deletion of such messages can be very costly and time-consuming, given the ever-increasing quantity of user-generated content online. In order to foster the development of offensive speech detection systems, more and more shared tasks have been organized on the detection and identification of offensive language in the past few years, covering several languages (Zampieri et al., 2019b; Basile et al., 2019; Wiegand et al., 2018; Bosco et al., 2018). In this paper, we present our submission to SemEval 2020 task 12, Multilingual Offensive Language Identification in Social Media (OffensEval 2) (Zampieri et al., 2020). We participated in subtask A, focused on the identification of offensive messages in 5 languages: English, Danish, Turkish, Arabic and Greek. In the SemEval 2019 edition of OffensEval, the task was focused on Offensive Language Identification in English (Zampieri et al., 2019a). The vast majority of the teams who obtained the highest macro-F1 scores in subtask A (focused on the binary identification of offensive language) used systems based on Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019). Since the 2020 edition of the task covers five languages, we aim at improving the performance of a multilingual pre-trained BERT system by using transfer learning. This is achieved by artificially creating more data for fine-tuning two parallel pre-trained BERT models by translating existing corpora in a different language and feeding the parallel data to a multiple input model. 2 Related Work The increased interest in the detection of offensive language on social media has entailed an increase in shared tasks on the topic (Wiegand et al., 2018; Bosco et al., 2018; Zampieri et al., 2019b), as well …

Content truncated to 500 words