XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages

Proceedings of the 28th International Conference on Computational Linguistics, pages 6350–6365 Barcelona, Spain (Online), December 8-13, 2020 6350 XHATE-999: Analyzing and Detecting Abusive Language Across Domains and Languages Goran Glavaš1∗, Mladen Karan2∗, Ivan Vuli´c3∗ 1Data and Web Science Group, University of Mannheim, Germany 2Text Analysis and Knowledge Engineering Lab, University of Zagreb, Croatia 3Language Technology Lab, TAL, University of Cambridge, UK goran@informatik.uni-mannheim.de mladen.karan@fer.hr iv250@cam.ac.uk Abstract We present XHATE-999, a multi-domain and multilingual evaluation data set for abusive language detection. By aligning test instances across six typologically diverse languages, XHATE-999 for the first time allows for disentanglement of the domain transfer and language transfer effects in abusive language detection. We conduct a series of domain- and language-transfer experiments with state-of-the-art monolingual and multilingual transformer models, setting strong baseline results and profiling XHATE-999 as a comprehensive evaluation resource for abusive language detection. Finally, we show that domain- and language-adaptation, via intermediate masked language modeling on abusive corpora in the target language, can lead to substantially improved abusive language detection in the target language in the zero-shot transfer setups. 1 Introduction In the era of ever-growing amounts of user-generated online content it is becoming increasingly difficult to scale up moderation efforts (Nobata et al., 2016). However, the need for moderation is rapidly increasing due to escalated toxic behavior online, enabled by “hiding” behind anonymous profiles, lack of physical contact between participants (i.e., the communication is then typically perceived as less personal), and lack of direct negative societal consequences (Perse and Lambe, 2016). Consequently, research on automated methods for detecting abusive language in user-generated content is becoming increasingly important. While such methods cannot completely replace human moderators, they are very helpful as assistance tools, offering moderation suggestions, thus partially automating and expediting human moderation work. The focus of abusive language detection is still predominantly on a single language – English, and single-domain setups (e.g., Twitter). However, some recent initiatives have aspired to broaden the scope of abusive detection methodology to other languages, showcasing the usefulness of cross-lingual transfer for the task (Sohn and Lee, 2019; Stappen et al., 2020; Pamungkas and Patti, 2019; Wiedemann et al., 2020, inter alia). Another line of research (Wiegand et al., 2018a; Karan and Šnajder, 2018; Waseem et al., 2018, inter alia) focuses on benefits of cross-domain transfer in monolingual settings. An interesting aspect, currently lacking in prior work, is the interaction of cross-lingual and cross-domain settings. Furthermore, except for some notable exceptions discussed in §2, previous work in cross-lingual setups is still tied to resource-rich and typologically similar languages (e.g., English, German, Spanish, Italian) (Stappen et al., 2020). We aim to fill both these gaps by introducing XHATE-999, a multilingual data set annotated for abusive language in three domains, and carefully manually translated from English to 5 typologically diverse languages, with 999 semantically aligned test instances across all languages. Unlike other abusive language detection data (surveyed in §2), XHATE-999 allows us to separate the effects that occur due to domain shift from the effects related to language shift. …

Content truncated to 500 words