KS@LTH at SemEval-2020 Task 12: Fine-tuning Multi- and Monolingual Transformer Models for Offensive Language Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 2045–2053 Barcelona, Spain (Online), December 12, 2020. 2045 KS@LTH at SemEval-2020 Task 12: Fine-tuning multi- and monolingual transformer models for offensive language detection Kasper Socha Lund University / Lund, Sweden fte10kso@student.lu.se Abstract This paper describes the KS@LTH system for SemEval-2020 Task 12 OffensEval2: Multilingual Offensive Language Identification in Social Media. We compare mono- and multilingual models based on fine-tuning pre-trained transformer models for offensive language identification in Arabic, Greek, English and Turkish. For Danish, we explore the possibility of fine-tuning a model pre-trained on a similar language, Swedish, and additionally also cross-lingual training together with English. Overall we find that monolingual models achieve higher macro-averaged F1 score. With cross-lingual training of Danish together with English, we achieve better results than by training on the small Danish dataset alone. For Arabic, Danish, English, Greek, and Turkish, we obtained macro-averaged F1 scores of 0.890, 0.775, 0.916, 0.848, and 0.810 ranking 6th, 5th, 6th, 3rd and 4th for each language, respectively. 1 Introduction Offensive language is a prevalent phenomenon in many online communities and social media platforms. Due to the vast amount of content, it is often infeasible to manually moderate all user submitted content. Computational methods for identifying this type of content is one possible way to help mitigate the problem. Different aspects of the problem such as aggression (Kumar et al., 2018), cyber bulling (Sprugnoli et al., 2018) and hate speech (Malmasi and Zampieri, 2017) have been studied in recent work. OffensEval 2019 used a new three-level hierarchical annotation schema to capture multiple aspects of offensive language in one framework (Zampieri et al., 2019a). While much of the previous work is focused on English, offensive language detection is a multilingual problem. Apart from country specific communities, large social media platforms such as Facebook and Twitter have many users interacting in their native tongue. Recently, offensive language detection addressed different languages such as German (Wiegand et al., 2018), Arabic (Mulki et al., 2019), Italian (Sanguinetti et al., 2018), and Spanish (Fersini et al., 2018). In OffensEval 2020, the first level task of offensive language detection has been expanded to cover five languages, Arabic, Danish, English, Greek, and Turkish. Transfer learning is nothing new in NLP but over time, the pre-training has become more complex, incorporating more context. In recent years, language models based on the transformer architecture pre-trained on large amounts of unlabeled text and then fine-tuned on downstream tasks have been used to achieve state-of-the-art (SOTA) results on many natural language benchmarks (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019). In OffensEval 2019, seven of the top ten models used BERT in some way (Zampieri et al., 2019b). One of the advantages of transfer learning is that it can potentially reduce the amount of labeled data that is needed. The model can learn general features of language from a large unannotated corpus during pre-training. Task specific features can then be learned from a smaller annotated corpus. On …

Content truncated to 500 words