Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-Based Hate

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1352 - 1368 July 10-15, 2022 ©2022 Association for Computational Linguistics HATEMOJI: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-Based Hate Hannah Rose Kirk1,2‡, Bertie Vidgen1,2, Paul Röttger1,2, Tristan Thrush3∗, Scott A. Hale1,2,4 1University of Oxford, 2The Alan Turing Institute, 3Hugging Face, 4Meedan ‡hannah.kirk@oii.ox.ac.uk Abstract Detecting online hate is a complex task, and low-performing models have harmful conse- quences when used for sensitive applications such as content moderation. Emoji-based hate is an emerging challenge for automated detec- tion. We present HATEMOJICHECK, a test suite of 3,930 short-form statements that allows us to evaluate performance on hateful language ex- pressed with emoji. Using the test suite, we ex- pose weaknesses in existing hate detection mod- els. To address these weaknesses, we create the HATEMOJIBUILD dataset using a human-and- model-in-the-loop approach. Models built with these 5,912 adversarial examples perform sub- stantially better at detecting emoji-based hate, while retaining strong performance on text-only hate. Both HATEMOJICHECK and HATEMOJI- BUILD are made publicly available.1 Content Warning This article contains exam- ples of hateful language from HATEMOJICHECK to illustrate its composition. Examples are quoted verbatim, except for slurs and profanity in text, for which the first vowel is replaced with an asterisk. The authors oppose the use of hateful language. 1 Introduction Online hate harms its targets, disrupts online com- munities, pollutes civic discourse and reinforces social power imbalances (Gelber and McNamara, 2016). The sheer scale of hateful content online has led to widespread use of automated detection systems to find, monitor and stop it (Vidgen et al., 2019; Gillespie, 2020). However, hateful content is complex and diverse, which makes it challenging to detection systems. One particular challenge is the use of emoji for expressing hate. Emoji are pic- torial representations that can be embedded in text, ∗TT’s work was done at Facebook AI Research. 1See our Github Repository. HATEMOJICHECK, HATEMOJIBUILD and the final HATEMOJI Model are also available on HuggingFace. allowing complex emotions, actions and intentions to be displayed concisely (Rodrigues et al., 2018). Over 95% of internet users have used an emoji and 10 million are sent every day (Brandwatch, 2018). Following England’s defeat in the Euro 2020 foot- ball final, there was widespread racist use of emoji such as , and (Jamieson, 2020). This pa- per focuses on emoji-based hate, answering two research questions: RQ1 What are the weaknesses of current detection systems for hate expressed with emoji? RQ2 To what extent can human-and-model-in-the- loop training improve the performance of de- tection systems for emoji-based hate? To answer RQ1, we present HATEMOJICHECK, a suite of functional tests for emoji-based hate. We provide 3,930 test cases for seven functionalities, covering six identities. 2,126 original test cases are matched with three types of challenging per- turbations to enable accurate evaluation of model decision boundaries (Gardner et al., 2020). We use HATEMOJICHECK to assess Google Jigsaw’s Per- spective API as well as …

Content truncated to 500 words