Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 943–949 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 943 Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection Michele Corazza†, Stefano Menini‡, Elena Cabrio§, Sara Tonelli‡, Serena Villata§ †University of Bologna, Italy ‡Fondazione Bruno Kessler, Trento, Italy §Universit´e Cˆote d’Azur, CNRS, Inria, I3S, France michele.corazza2@unibo.it {menini,satonelli}@fbk.eu {elena.cabrio,serena.villata}@univ-cotedazur.fr Abstract Recent studies have demonstrated the effec- tiveness of cross-lingual language model pre- training on different NLP tasks, such as nat- ural language inference and machine trans- lation. In our work, we test this approach on social media data, which are particularly challenging to process within this framework, since the limited length of the textual messages and the irregularity of the language make it harder to learn meaningful encodings. More specifically, we propose a hybrid emoji-based Masked Language Model (MLM) to leverage the common information conveyed by emo- jis across different languages and improve the learned cross-lingual representation of short text messages, with the goal to perform zero- shot abusive language detection. We compare the results obtained with the original MLM to the ones obtained by our method, showing im- proved performance on German, Italian and Spanish. 1 Introduction The extensive use of large-scale self-supervised pre- training has greatly contributed to recent progress in many Natural Language Processing (NLP) tasks (Devlin et al., 2019; Liu et al., 2019; Con- neau and Lample, 2019). In this context, masked language modelling objectives represent one of the main novelties of these approaches, where some tokens of an input sequence are randomly masked, and the objective is to predict these masked posi- tions taking the corrupted sequence as input. Still, little attention has been devoted to the adaptation of these techniques to tasks dealing with social media data, probably because they are character- ized by a very domain-specific language, with high variability and instability. Nevertheless, all these challenges make social media data an inter- esting testbed for novel deep-learning architectures, around the research question: how could the mask- ing mechanism be adapted to target social media language? In this paper, we address the above issue by adapting a novel architecture for cross-lingual mod- els called XLM (Conneau and Lample, 2019) to zero-shot abusive language detection, a task that has gained increasing importance given the recent surge in abusive online behavior and the need to develop reliable and efficient methods to detect it. In particular, we evaluate two methods to pre-train bilingual language models, one similar to the origi- nal XLM masked model, and the other based on a novel hybrid emoji-based masked model. We then evaluate them on zero-shot abusive language detec- tion for Italian, German and Spanish, showing that, although our results are below the state-of-the-art in a monolingual setting, the proposed solutions to adapt XLM to social media data are beneficial and can be effectively extended to other languages. In the following, Section 2 discusses the related work. Section 3 describes our approach to train cross-lingual models for social media data …

Content truncated to 500 words