Improving the Detection of Multilingual Online Attacks with Rich Social Media Data from Singapore

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12705–12721 July 9-14, 2023 ©2023 Association for Computational Linguistics Improving the Detection of Multilingual Online Attacks with Rich Social Media Data from Singapore Janosch Haber1*, Bertie Vidgen2*, Matt Chapman*, Vibhor Agarwal3*, Roy Ka-Wei Lee4, Yong Keong Yap5, and Paul Röttger6* 1Queen Mary University London 2Alan Turing Institute 3University of Surrey 4Singapore Univ. of Technology & Design 5DSO National Laboratories 6Univ. of Oxford Abstract Toxic content is a global problem, but most resources for detecting toxic content are in En- glish. When datasets are created in other lan- guages, they often focus exclusively on one language or dialect. In many cultural and ge- ographical settings, however, it is common to code-mix languages, combining and inter- changing them throughout conversations. To shine a light on this practice, and enable more research into code-mixed toxic content, we in- troduce SOA, a new multilingual dataset of on- line attacks. Using the multilingual city-state of Singapore as a starting point, we collect a large corpus of Reddit comments in Indone- sian, Malay, Singlish, and other languages, and provide fine-grained hierarchical labels for on- line attacks. We publish the corpus with rich metadata, as well as additional unlabelled data for domain adaptation. We share comprehen- sive baseline results, show how the metadata can be used for granular error analysis, and demonstrate the benefits of domain adaptation for detecting multilingual online attacks. Content warning: This article contains illus- trative examples of toxic content. 1 Introduction Toxic content, such as hate speech and abuse, is a global problem, but most resources for detecting toxic content are in English (Vidgen and Derczyn- ski, 2020; Poletto et al., 2021; Röttger et al., 2022a). This makes it difficult to develop effective models for detecting toxic content in other languages, and as a consequence, non-English speakers across the world are less protected against toxic content. New datasets and models for non-English lan- guages often focus exclusively on one language or dialect. In many cultural and geographical set- tings, however, languages and dialects are often code-mixed, i.e., combined or used interchange- ably within a conversation or even a single utter- *Work completed at Rewire. ance (Gibbons, 1987; Rijhwani et al., 2017). So far, this practice has received very limited atten- tion in toxic content research, with most work on code-mixed content focusing on Hinglish, which is a mix of Hindi and English (e.g. Mathur et al., 2018a; Bohra et al., 2018; Sengupta et al., 2022). In this article, we take a step towards address- ing this issue by introducing SOA, a new multilin- gual dataset of Singapore-centered online attacks. Singapore is a multilingual city-state in Southeast Asia, with five million inhabitants from a wide range of ethnic, religious, and cultural backgrounds. Singapore’s official languages are English, Malay, Singaporean Mandarin and Tamil, but many other languages are widely spoken, including the code- mixed Singlish language and Indonesian, which is closely related to Malay. Using the r/Singapore sub- …

Content truncated to 500 words