ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3309 - 3326 May 22-27, 2022 c⃝2022 Association for Computational Linguistics TOXIGEN: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection Warning: this paper discusses and contains content that can be offensive or upsetting. Thomas Hartvigsen ♠ Saadia Gabriel♥ Hamid Palangi♣ Maarten Sap▲△ Dipankar Ray♦ Ece Kamar♣ ♠Massachusetts Institute of Technology ♥University of Washington ♣Microsoft Research ▲Allen Institute for AI △Carnegie Mellon University ♦Microsoft tomh@mit.edu, skgabrie@cs.washington.edu, hpalangi@microsoft.com, maartensap@cmu.edu {diray,eckamar}@microsoft.com Abstract Toxic language detection systems often falsely flag text that contains minority group men- tions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic lan- guage. To help mitigate these issues, we cre- ate TOXIGEN, a new large-scale and machine- generated dataset of 274k toxic and benign statements about 13 minority groups. We de- velop a demonstration-based prompting frame- work and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model (Brown et al., 2020). Controlling ma- chine generation in this way allows TOXIGEN to cover implicitly toxic text at a larger scale, and about more demographic groups, than pre- vious resources of human-written text. We conduct a human evaluation on a challeng- ing subset of TOXIGEN and find that annota- tors struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data im- proves its performance on human-written data substantially. We also demonstrate that TOXI- GEN can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset. 1 Introduction Toxic language detectors often over-rely on minor- ity identity mentions1 when flagging a statement as toxic, without considering the deeper seman- tic meaning of the statement (Dixon et al., 2018; Röttger et al., 2021). This can lead to severe under- detection of subtle hate (e.g., “They have been bred 1In this work, we use “minority” to refer to social and demographic groups that are frequently the targets of oppres- sion, discrimination, or prejudice (RWJF, 2017), from a U.S. socio-cultural perspective. to be good at sports and entertainment, but not much else”; Figure 1) and over-detection of benign statements (e.g., “child abuse is wrong, racism is wrong, sexism is wrong”; Figure 1). Impor- tantly, such biases in toxicity detection risk further marginalizing or censoring minority groups (Yasin, 2018; Sap et al., 2019; Dias Oliva et al., 2020; Are, 2020; Díaz and Hecht-Felella, 2021). We introduce TOXIGEN, a large-scale machine- generated dataset of 274,186 toxic and benign state- ments. To create this dataset, we leverage the massive pretrained language model GPT-3 (Brown et al., 2020), which is known to produce close- to-human-like text (Clark et al., 2021; Dou et al., 2021) but also easily generates socially …

Content truncated to 500 words