The Risk of Racial Bias in Hate Speech Detection

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1668 The Risk of Racial Bias in Hate Speech Detection Maarten Sap♦ Dallas Card♣ Saadia Gabriel♦ Yejin Choi♦♥ Noah A. Smith♦♥ ♦Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, USA ♣Machine Learning Department, Carnegie Mellon University, Pittsburgh, USA ♥Allen Institute for Artificial Intelligence, Seattle, USA msap@cs.washington.edu Abstract We investigate how annotators’ insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection mod- els, potentially amplifying harm against mi- nority populations. We first uncover unex- pected correlations between surface markers of African American English (AAE) and rat- ings of toxicity in several widely-used hate speech datasets. Then, we show that models trained on these corpora acquire and propagate these biases, such that AAE tweets and tweets by self-identified African Americans are up to two times more likely to be labelled as of- fensive compared to others. Finally, we pro- pose dialect and race priming as ways to re- duce the racial bias in annotation, showing that when annotators are made explicitly aware of an AAE tweet’s dialect they are significantly less likely to label the tweet as offensive. 1 Introduction Toxic language (e.g., hate speech, abusive speech, or other offensive speech) primarily targets mem- bers of minority groups and can catalyze real- life violence towards them (O’Keeffe et al., 2011; Cleland, 2014; Mozur, 2018). Social media platforms are under increasing pressure to re- spond (Trindade, 2018), but automated removal of such content risks further suppressing already- marginalized voices (Yasin, 2018; Dixon et al., 2018). Thus, great care is needed when develop- ing automatic toxic language identification tools. The task is especially challenging because what is considered toxic inherently depends on social context (e.g., speaker’s identity or dialect). In- deed, terms previously used to disparage com- munities (e.g., “n*gga”, “queer”) have been re- claimed by those communities while remaining offensive when used by outsiders (Rahman, 2012). Figure 1 illustrates how phrases in the African American English dialect (AAE) are labelled by a publicly available toxicity detection tool as much crowdsourcing PerspectiveAPI Toxicity score I saw him yesterday. What's up, bro! I saw his ass yesterday. 95% 6% Wussup, n*gga! 90% 7% Wussup, n*gga! classifier Non-toxic tweets (per Spears, 1998) Figure 1: Phrases in African American English (AAE), their non-AAE equivalents (from Spears, 1998), and toxicity scores from PerspectiveAPI.com. Per- spective is a tool from Jigsaw/Alphabet that uses a convolutional neural network to detect toxic language, trained on crowdsourced data where annotators were asked to label the toxicity of text without metadata. more toxic than general American English equiv- alents, despite their being understood as non-toxic by AAE speakers (Spears, 1998, see §2). In this work, we first empirically characterize the racial bias present in several widely used Twit- ter corpora annotated for toxic content, and quan- tify the propagation of this bias through models trained …

Content truncated to 500 words