Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2636–2651 May 2-6, 2023 ©2023 Association for Computational Linguistics Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro NVIDIA {sprabhumoye, mpatwary, mshoeybi, bcatanzaro}@nvidia.com Abstract Pretrained large language models have become indispensable for solving various natural lan- guage processing (NLP) tasks. However, safely deploying them in real world applications is challenging because they generate toxic con- tent. To address this challenge, we propose two novel pretraining data augmentation strategies that significantly reduce model toxicity with- out compromising its utility. Our two strate- gies are: (1) MEDA: adds raw toxicity score as meta-data to the pretraining samples, and (2) INST: adds instructions to those samples indi- cating their toxicity. Our results indicate that our best performing strategy (INST) substan- tially reduces the toxicity probability up to 61% while preserving the accuracy on five bench- mark NLP tasks as well as improving AUC scores on four bias detection tasks by 1.3%. We also demonstrate the generalizability of our techniques by scaling the number of training samples and the number of model parameters. 1 Introduction Pretrained large language models (LMs) have become indispensable for solving various NLP tasks (Brown et al., 2020; Smith et al., 2022; Chowdhery et al., 2022), yet it has been challeng- ing to safely deploy them for real world applica- tions (McGuffie and Newhouse, 2020; Prabhumoye et al., 2021a). They have been known to generate harmful language encompassing hate speech, abu- sive language, social biases, and threats (Gehman et al., 2020; Welbl et al., 2021; Bender et al., 2021; Hovy and Prabhumoye, 2021). These harmful gen- erations are broadly referred to as “toxicity”.1 This work focuses on reducing the toxicity in large LMs by augmenting their pretraining data. Prior work has primarily focused on reducing toxicity either by finetuning LMs on non-toxic 1In this work we use toxicity as defined by Perspec- tiveAPI (PerspectiveAPI, 2022) but our techniques can be applied to other broader definitions of bias or toxicity. Figure 1: Overview of the proposed approaches and the baseline (BASE). We propose two new data augmen- tation strategies, MEDA and INST. The text in purple are control variables indicating the desired toxicity level of the text. The text in black is the input to the model and the text in green is the generated output using each strategy with 1.3b parameter model. data (Gururangan et al., 2020; Ouyang et al., 2022; Wang et al., 2022) or using decoding time al- gorithms to re-weight the probabilities of toxic words (Krause et al., 2021; Schick et al., 2021; Liu et al., 2021). These methods typically incur further costs of finetuning additional LMs (Krause et al., 2021; Liu et al., 2021), generating large amount of non-toxic data (Wang et al., 2022), or procur- ing human feedback (Ouyang et al., 2022). These techniques are known to reduce toxicity but also compromise perplexity and downstream task per- formance …

Content truncated to 500 words