RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3356 REALTOXICITYPROMPTS: Evaluating Neural Toxic Degeneration in Language Models Samuel Gehman⇧ Suchin Gururangan⇧† Maarten Sap⇧ Yejin Choi⇧† Noah A. Smith⇧† ⇧Paul G. Allen School of Computer Science & Engineering, University of Washington †Allen Institute for Artificial Intelligence Seattle, USA {sgehman,sg01,msap,yejin,nasmith}@cs.washington.edu Abstract Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deploy- ment. We investigate the extent to which pre- trained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RE- ALTOXICITYPROMPTS, a dataset of 100K nat- urally occurring, sentence-level prompts de- rived from a large corpus of English web text, paired with toxicity scores from a widely- used toxicity classifier. Using REALTOXICI- TYPROMPTS, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess sev- eral controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxic- ity than simpler solutions (e.g., banning “bad” words), no current method is failsafe against neural toxic degeneration. To pinpoint the po- tential cause of such persistent toxic degenera- tion, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Rad- ford et al., 2019), and find a significant amount of offensive, factually unreliable, and other- wise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining. 1 Introduction Although they are the backbone of many modern NLP systems (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019), language models (LMs) pretrained on large web text corpora suffer from degenerate and biased behavior (Sheng et al., 2019; Wallace et al., 2019). As illustrated in Figure 1, they can easily degenerate into toxicity, even with- out explicitly toxic prompts, which hinders their ������������������������� �������������������� ����������������������� ������������������������ �� �� �� �� �� �� �� �� ������������������� �������������� ��������������� ������������������ �������������������� �������������������� �� ��� ��� ��� Figure 1: Non-toxic examples from REALTOXICI- TYPROMPTS, a new testbed for evaluating neural gen- erations and their toxicity. Despite not containing any toxic language as measured by PERSPECTIVE API, these prompts cause several pretrained LMs to system- atically generate highly toxic text (shown in Table 17 in Appendix §E). safe deployment (McGuffie and Newhouse, 2020). We first introduce a framework to systemat- ically measure the risk of toxic degeneration by pretrained LMs. We release REALTOXICI- TYPROMPTS (§4), a set of 100K naturally oc- curring prompts (i.e., sentence prefixes; Figure 1) extracted from a large corpus of English web text and paired with toxicity scores from a widely used and commercially deployed toxicity detector (PERSPECTIVE API). We show that popular LMs produce toxic generations when conditioned on our prompts, even those that are …

Content truncated to 500 words