Language Detoxification with Attribute-Discriminative Latent Space

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10149–10171 July 9-14, 2023 ©2023 Association for Computational Linguistics Language Detoxification with Attribute-Discriminative Latent Space Jin Myung Kwak1∗, Minseon Kim1∗, Sung Ju Hwang1,2 KAIST1, DeepAuto2 {kwak.jinmyung, minseonkim, sjhwang82}@kaist.ac.kr Abstract Transformer-based Language Models (LMs) have achieved impressive results on natural lan- guage understanding tasks, but they can also generate toxic text such as insults, threats, and profanity, limiting their real-world applications. To overcome this issue, a few text generation approaches aim to detoxify toxic texts using additional LMs or perturbations. However, pre- vious methods require excessive memory, com- putations, and time which are serious bottle- necks in their real-world application. To ad- dress such limitations, we propose an effec- tive yet efficient method for language detox- ification using an attribute-discriminative la- tent space. Specifically, we project the latent space of an original Transformer LM onto a discriminative latent space that well-separates texts by their attributes using a projection block and an attribute discriminator. This allows the LM to control the text generation to be non- toxic with minimal memory and computation overhead. We validate our model, Attribute- Discriminative Language Model (ADLM) on detoxified language and dialogue generation tasks, on which our method significantly out- performs baselines both in performance and efficiency. 1 Introduction Pre-training language models (LMs) on large-scale web text corpora (i.e., Common Crawl and Open- WebTextCorpus (Gokaslan and Cohen, 2019)) has significantly improved their language generation performances (Radford et al., 2019; Yang et al., 2019; Dai et al., 2019; Shoeybi et al., 2019; Li et al., 2020; Brown et al., 2020), by allowing them to learn meaningful relations between words. How- ever, since the models are trained on massive web- crawled text data which is not exhaustively filtered, * Equal contribution; ordering determined by coin toss Warning: this paper contains offensive or upsetting examples. PPLM GPT2 DAPT GeDi DExperts ADLM Exp. Max Toxicity 0.3 0.4 0.5 0.6 0.7 0.8 Parameter (M) 0 500 1000 GPT2 DAPT GeDi DExperts ADLM PPLM Exp. Max Toxicity 0.3 0.4 0.5 0.6 0.7 0.8 Time (s) 10 100 Figure 1: Memory and computational efficiency vs. Exp. Max Toxicity. Comparison of toxicity of the gen- erated texts between previous language detoxification methods and ours, on the number of model parameters and inference time per 100 generated texts with a single GPU. Toxicity is calculated on random-10K prompts from RealToxicityPrompts (Gehman et al., 2020). Our model achieves the best language detoxification perfor- mance while being time- and memory- efficient. they are prone to generating unexpected and unde- sired texts (Sheng et al., 2019; Wallace et al., 2019) which are often also inappropriate (See Table 1). Specifically, LMs trained on unfiltered texts can randomly generate racial slurs, sexually ex- plicit and violent expressions, which are highly toxic (Groenwold et al., 2020; Luccioni and Vi- viano, 2021; Xu et al., 2021; Dale et al., 2021). This is one of the main obstacles in deploying pre- trained LMs to real-world applications (e.g., …

Content truncated to 500 words