DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6691–6706 August 1–6, 2021. ©2021 Association for Computational Linguistics 6691 DEXPERTS: Decoding-Time Controlled Text Generation with Experts and Anti-Experts Alisa Liu♥ Maarten Sap♥ Ximing Lu♥♣ Swabha Swayamdipta♣ Chandra Bhagavatula♣ Noah A. Smith♥♣ Yejin Choi♥♣ ♥Paul G. Allen School of Computer Science & Engineering, University of Washington ♣Allen Institute for Artificial Intelligence alisaliu@cs.washington.edu Abstract Despite recent advances in natural language generation, it remains challenging to control attributes of generated text. We propose DEX- PERTS: Decoding-time Experts, a decoding- time method for controlled text generation that combines a pretrained language model with “expert” LMs and/or “anti-expert” LMs in a product of experts. Intuitively, under the ensemble, tokens only get high probabil- ity if they are considered likely by the ex- perts and unlikely by the anti-experts. We ap- ply DEXPERTS to language detoxification and sentiment-controlled generation, where we outperform existing controllable generation methods on both automatic and human evalua- tions. Moreover, because DEXPERTS operates only on the output of the pretrained LM, it is effective with (anti-)experts of smaller size, in- cluding when operating on GPT-3. Our work highlights the promise of tuning small LMs on text with (un)desirable attributes for efficient decoding-time steering. 1 Introduction Controlling the output of pretrained language mod- els (LMs) is crucial for achieving useful and safe language generation applications, such as non- offensive sentence completion or friendly conversa- tion generation (See et al., 2019; Sheng et al., 2020; Gehman et al., 2020). For example, a safe comple- tion to the prompt “When she rejected his advance, he grabbed...” requires avoiding word choices that could lead to continuations with gender-based vio- lence (e.g., “her”; Figure 1). Without such steering, these language models risk generating mindless and offensive content (Sheng et al., 2019; Holtzman et al., 2020) which hinders their safe deployment (Brockman et al., 2020; Bender et al., 2021). Importantly, as the scale of pretrained LMs increases (e.g., 175B and 1.6T parameters; Brown et al., 2020; Fedus et al., Figure 1: Illustration of DEXPERTS, where a toxic LM acts as an “anti-expert” and a non-toxic LM acts as an “expert”. In this toy example, given the prompt, “When she rejected his advance, he grabbed,” the toxic LM assigns greater weight to “her” than “his”, expressing subtle signals of toxicity that can be leveraged for effec- tive attribute control. The difference in logits z` ´ z´ output by the expert and anti-expert represents the per- turbations to make to the logits z of the pretrained “base” LM. 2021), finetuning or re-training approaches are be- coming increasingly computationally infeasible for most researchers. We propose DEXPERTS,1 a decoding-time method for controlled text generation based on a 1DEXPERTS stands for Decoding-time Experts. Our code is available at alisawuffles/DExperts.https://github.com/ 6692 product of experts (Hinton, 2002). Our method combines an out-of-the-box pretrained (“base”) LM with “expert” LMs and/or “anti-expert” LMs, which model text with desirable and undesirable at- tributes, respectively. …

Content truncated to 500 words