GruPaTo at SemEval-2020 Task 12: Retraining mBERT on Social Media and Fine-tuned Offensive Language Models

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1546–1554 Barcelona, Spain (Online), December 12, 2020. 1546 GruPaTo at SemEval-2020 Task 12: Retraining mBERT on Social Media and Fine-tuned Offensive Language Models Davide Colla♣⋄, Tommaso Caselli♣, Valerio Basile⋄, Jelena Mitrovi´c‡, Michael Granitzer‡ ♣University of Groningen, ⋄University of Turin, ‡University of Passau Groningen The Netherlands, Turin Italy, Passau Germany ⋄{valerio.basile|davide.colla}@unito.it, ♣t.caselli@rug.nl ‡jelena.mitrovic|michael.granitzer}@uni-passau.de Abstract We introduce an approach to multilingual Offensive Language Detection based on the mBERT transformer model. We download extra training data from Twitter in English, Danish, and Turkish, and use it to re-train the model. We then fine-tuned the model on the provided training data and, in some configurations, implement transfer learning approach exploiting the typological relatedness between English and Danish. Our systems obtained good results across the three languages (.9036 for EN, .7619 for DA, and .7789 for TR). 1 Introduction The growth of Social Media has seen the spread of two different but connected phenomena: on the one hand, they helped to create a more open and connected world, and, on the other hand, they contributed to the spread of offensive and abusive behaviors. Although the use of “bad language” is intimately connected with freedom of speech, the phenomenon has become so pervasive that developing Natural Language Processing (NLP) systems that automatically and efficiently detect and classify offensive on-line content is a pressing need (Nobata et al., 2016; Kennedy et al., 2017). SemEval-2020 Task 12: OffensEval 2 (Zampieri et al., 2020) is a follow-up edition of SemEval-2019 Task 6: OffensEval (Zampieri et al., 2019a) and it addresses the problem of offensive language detection in Twitter messages by focusing on two pending issues: multilingualism and hierarchical tagset annotation. The multilingualism issue is targeted by providing for the first time data in 5 different languages, namely English, Danish, Greek, Turkish, and Arabic, by applying a shared definition of offensive language. The languages cover different values of the typological spectrum in terms of type (Fusional vs. Agglutinative), language family (Indo-European vs. Altaic vs. Afro-African), genus (Germanic vs. Greek vs. Turkic vs. Semitic), Subject-Object-Verb 1 order (SVO vs. no dominant order vs. SOV vs. VSO) (Dryer and Haspelmath, 2013; Ramat and Baldry, 2011), as well as writing systems. The multilingual aspect poses two additional challenges: (i.) availability of NLP tools and language resources as some of the proposed languages are considered low-resourced (e.g., Danish and Greek) (Rehm and Uszkoreit, 2013); and (ii.) differences in the perceived offensiveness of a message. In particular, given that offensiveness is a highly subjective category, seeing that a message is always “offensive for someone” (Vidgen et al., 2019), different communities of speakers may have different perceptions of what is offensive or not. The use of a shared definition is a way of mitigating the potential differences across communities, but this aspect cannot be ignored in the development of a system for offensive language detection. The hierarchical annotation tagset is reflected in three sub-tasks, namely: • Sub-task A: Offensive language identification: The task consists in predicting …

Content truncated to 500 words