Detoxifying Language Models Risks Marginalizing Minority Voices

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397 June 6–11, 2021. ©2021 Association for Computational Linguistics 2390 Detoxifying Language Models Risks Marginalizing Minority Voices Albert Xu♦ Eshaan Pathak♦ Eric Wallace♦ Suchin Gururangan♠ Maarten Sap♠ Dan Klein♦ ♦UC Berkeley ♠University of Washington {albertxu3, eshaanpathak, ericwallace, klein}@berkeley.edu {sg01, msap}@cs.washington.edu Abstract Language models (LMs) must be both safe and equitable to be responsibly deployed in prac- tice. With safety in mind, numerous detoxifi- cation techniques (e.g., Dathathri et al. 2020; Krause et al. 2020) have been proposed to mit- igate toxic LM generations. In this work, we show that these detoxification techniques hurt equity: they decrease the utility of LMs on language used by marginalized groups (e.g., African-American English and minority iden- tity mentions). In particular, we perform au- tomatic and human evaluations of text genera- tion quality when LMs are conditioned on in- puts with different dialects and group identi- fiers. We find that detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups. We identify that these failures stem from detoxi- fication methods exploiting spurious correla- tions in toxicity datasets. Overall, our results highlight the tension between the controllabil- ity and distributional robustness of LMs. 1 Introduction Recent neural language models (LMs) have shown enormous improvements in text generation abili- ties. A key factor behind these improvements is large training corpora that are collected from on- line sources (Radford et al., 2019). Unfortunately, because such corpora are too large to filter granu- larly (Roller et al., 2020), they inevitably contain so-called toxic examples: undesirable language such as expletives, slurs, or other offensive and threatening speech. When trained on such data, LMs inevitably learn to generate toxic text (Hen- derson et al., 2018; Wallace et al., 2019). To address this issue, recent work has turned towards detoxifying LMs: reducing toxic gener- ations without affecting perplexity or generation quality on nontoxic inputs. Existing detoxifica- tion strategies involve techniques such as finetun- ing LMs on nontoxic data (Gehman et al., 2020) or incorporating a toxicity discriminator during de- coding (Dathathri et al., 2020). Our evaluation of these techniques shows that they are indeed effec- tive at mitigating toxicity, but at what cost? We demonstrate that detoxification can hurt LM utility on language used by minority groups. Concretely, we evaluate detoxified LMs on text with minority identity mentions (e.g., words such as “gay” or “Muslim”) and surface markers of African-American English (Green, 2002, AAE). We first show that, compared to text contain- ing White-Aligned English (WAE), detoxification causes a disproportionately large increase in LM perplexity on text with AAE and minority iden- tity mentions. Moreover, increasing the strength of detoxification amplifies this bias. The same trends hold when evaluating the text generation quality of LMs using crowdworkers. When conditioned on WAE text, detoxified LMs can roughly maintain the topic, fluency, and style of an input prompt. However, generation quality deteriorates when models are conditioned on AAE text, …

Content truncated to 500 words