Hate Speech Classifiers Learn Normative Social Stereotypes

Hate Speech Classifiers Learn Normative Social Stereotypes Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, Morteza Dehghani University of Southern California, USA {mostafaz,atari,btkenned,mdehghan}@usc.edu Abstract Social stereotypes negatively impact individ- uals’ judgments about different groups and may have a critical role in understanding lan- guage directed toward marginalized groups. Here, we assess the role of social stereotypes in the automated detection of hate speech in the English language by examining the impact of social stereotypes on annotation behaviors, annotated datasets, and hate speech classifiers. Specifically, we first investigate the impact of novice annotators’ stereotypes on their hate-speech-annotation behavior. Then, we examine the effect of normative stereo- types in language on the aggregated annota- tors’ judgments in a large annotated corpus. Finally, we demonstrate how normative stereo- types embedded in language resources are associated with systematic prediction errors in a hate-speech classifier. The results dem- onstrate that hate-speech classifiers reflect so- cial stereotypes against marginalized groups, which can perpetuate social inequalities when propagated at scale. This framework, combin- ing social-psychological and computational- linguistic methods, provides insights into sources of bias in hate-speech moderation, informing ongoing debates regarding machine learning fairness. Introduction Artificial Intelligence (AI) technologies are prone to acquiring cultural, social, and institutional bi- ases from the real-world data on which they are trained (McCradden et al., 2020; Mehrabi et al., 2021; Obermeyer et al., 2019). AI models trained on biased datasets both reflect and amplify those biases (Crawford, 2017). For example, the domi- nant practice in modern Natural Language Pro- cessing (NLP)—which is to train AI systems on large corpora of human-generated text data— leads to representational biases, such as preferring European American names over African Amer- ican names (Caliskan et al., 2017), associating words with more negative sentiment with phrases referencing persons with disabilities (Hutchinson et al., 2020), making ethnic stereotypes by asso- ciating Hispanics with housekeepers and Asians with professors (Garg et al., 2018), and assign- ing men to computer programming and women to homemaking (Bolukbasi et al., 2016). Moreover, NLP models are particularly suscep- tible to amplifying biases when their task involves evaluating language generated by or describing a social group (Blodgett and O’Connor, 2017). For example, previous research has shown that tox- icity detection models associate documents con- taining features of African American English with higher offensiveness than text without those fea- tures (Sap et al., 2019; Davidson et al., 2019). Similarly, Dixon et al. (2018) demonstrate that models trained on social media posts are prone to erroneously classifying ‘‘I am gay’’ as hate speech. Therefore, using such models for moder- ating social-media platforms can yield dispropor- tionate removal of social-media posts generated by or mentioning marginalized groups (Davidson et al., 2019). This unfair assessment negatively impacts marginalized groups’ representation in online platforms, which leads to disparate impacts on historically excluded groups (Feldman et al., 2015). Mitigating biases in hate speech detection, nec- essary for viable automated content moderation (Davidson et al., 2017; Mozafari et al., 2020), has recently gained momentum (Davidson et al., 2019; Dixon et al., …

Content truncated to 500 words