Large-Scale Hate Speech Detection with Cross-Domain Transfer

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2215–2225 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2215 Large-Scale Hate Speech Detection with Cross-Domain Transfer Cagri Toraman, Furkan S¸ahinuc¸, Eyup Halit Yilmaz Aselsan Research Center, Ankara, Turkey {ctoraman, fsahinuc, ehyilmaz}@aselsan.com.tr Abstract The performance of hate speech detection models relies on the datasets on which the models are trained. Existing datasets are mostly prepared with a limited number of instances or hate domains that define hate topics. This hinders large-scale analysis and transfer learning with respect to hate domains. In this study, we construct large-scale tweet datasets for hate speech detection in English and a low-resource language, Turkish, consisting of human-labeled 100k tweets per each. Our datasets are designed to have equal number of tweets distributed over five domains. The experimental results supported by statistical tests show that Transformer-based language models outperform conventional bag-of-words and neural models by at least 5% in English and 10% in Turkish for large-scale hate speech detection. The performance is also scalable to different training sizes, such that 98% of performance in English, and 97% in Turkish, are recovered when 20% of training instances are used. We further examine the generalization ability of cross-domain transfer among hate domains. We show that 96% of the performance of a target domain in average is recovered by other domains for English, and 92% for Turkish. Gender and religion are more successful to generalize to other domains, while sports fail most. Keywords: cross-domain transfer, hate speech detection, low-resource language, offensive language, scalability 1. Introduction With the growth of social media platforms, hate speech towards people who do not share the same identity or community becomes more visible (Twitter, 2021). Consequences of online hate speech can be real-life vi- olence against other people and communities (Byman, 2021). The need to automatically detect hate speech text is thereby urging. Existing solutions to detect hate speech mostly rely on supervised scheme, resulting in a strict dependency on the quality and quantity of labeled data. Most of the datasets labeled by experts for hate speech detection are not large in size due to the labor cost (Poletto et al., 2021), causing a lack of detailed experiments on model generalization and scalability. Indeed, most studies on hate speech detection report high performances on their test sets, while their generalization capabilities to other datasets can be limited (Arango et al., 2019). More- over, existing datasets for hate speech detection are very limited for low-resource languages such as Turkic languages (Poletto et al., 2021). We thereby construct large-scale datasets for hate speech detection in English and Turkish, consisting of human-labeled 100k tweets per each and compare the performance of state-of-the- art models on these large-scale datasets. Hateful language can be expressed in various topics (we refer to topics as hate domains). Hate domains vary depending on the target group. For instance, misogyny is an example of the domain of gender-based hatred. Existing studies mostly do not consider …

Content truncated to 500 words