Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884 - 5906 July 10-15, 2022 ©2022 Association for Computational Linguistics Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection Maarten Sap♡♠ Swabha Swayamdipta♠ Laura Vianna♢ Xuhui Zhou♣ Yejin Choi♡♠ Noah A. Smith♡♠ ♡Paul G. Allen School of Computer Science, University of Washington, Seattle, WA, USA ♠Allen Institute for AI, Seattle, WA, USA ♢Department of Psychology, University of Washington, Seattle, WA, USA ♣Georgia Institute of Technology, Atlanta, GA, USA {maartens,swabhas}@allenai.org Abstract Warning: this paper discusses and contains content that is offensive or upsetting. The perceived toxicity of language can vary based on someone’s identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases. We seek to understand the who, why, and what behind biases in toxic- ity annotations. In two online studies with demographically and politically diverse par- ticipants, we investigate the effect of annota- tor identities (who) and beliefs (why), draw- ing from social psychology research about hate speech, free speech, racist beliefs, po- litical leaning, and more. We disentangle what is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity. Notably, more conservative annotators and those who scored highly on our scale for racist beliefs were less likely to rate anti-Black language as toxic, but more likely to rate AAE as toxic. We additionally present a case study illustrat- ing how a popular toxicity detection system’s ratings inherently reflect only specific beliefs and perspectives. Our findings call for con- textualizing toxicity labels in social variables, which raises immense implications for toxic language annotation and detection. 1 Introduction Determining whether a text is toxic (i.e., con- tains hate speech, abuse, or is offensive) is inher- ently a subjective task that requires a nuanced un- derstanding of the pragmatic implications of lan- guage (Fiske, 1993; Croom, 2011; Waseem et al., 2021). Without this nuance, both humans and machines are prone to biased judgments, such as over-relying on seemingly toxic keywords (e.g., expletives, swearwords; Dinan et al., 2019; Han and Tsvetkov, 2020) or backfiring against minori- ties (Yasin, 2018; Are, 2020, i.a.). For exam- ple, racial biases have been uncovered in toxic language detection where text written in African American English (AAE) is falsely flagged as toxic (Sap et al., 2019; Davidson et al., 2019). The crux of the issue is that not all text is equally toxic for everyone (Waseem, 2016; Al Kuwatly et al., 2020). Yet, most previous re- search has treated this detection as a simple clas- sification with one correct label, obtained by av- eraging judgments by a small set of human raters per post (Waseem and Hovy, 2016; Wulczyn et al., 2017; Davidson et al., 2017; Founta et al., 2018; Zampieri et al., 2019). Such approaches ignore the variance …

Content truncated to 500 words