Gradient-based Constrained Sampling from Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2251 - 2277 December 7-11, 2022 ©2022 Association for Computational Linguistics Gradient-Based Constrained Sampling from Language Models Sachin Kumar♣ Biswajit Paria♡ Yulia Tsvetkov♠ ♣Language Technologies Institute, Carnegie Mellon University, Pittsburgh PA ♡Machine Learning Department, Carnegie Mellon University, Pittsburgh PA ♠Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA {sachink,bparia}@cs.cmu.edu, yuliats@cs.washington.edu Abstract Large pretrained language models generate flu- ent text but are notoriously hard to control- lably sample from. In this work, we study con- strained sampling from such language models: generating text that satisfies user-defined con- straints, while maintaining fluency and model’s performance in a downstream task. We propose MUCOLA—a sampling procedure that com- bines the log-likelihood of the language model with arbitrary (differentiable) constraints in a single energy function, and then generates sam- ples in a non-autoregressive manner. Specifi- cally, it initializes the entire output sequence with noise and follows a Markov chain defined by Langevin Dynamics using the gradients of the energy function. We evaluate MUCOLA on text generation with soft and hard constraints as well as their combinations obtaining signifi- cant improvements over competitive baselines for toxicity avoidance, sentiment control, and keyword-guided generation.1 1 Introduction Transformer-based language models (LMs) trained on web-scale corpora (Radford et al., 2019; Raf- fel et al., 2020; Brown et al., 2020) are generat- ing impressively realistic texts. Despite having human-level fluency, they are far from reaching human-level communication abilities and are hard to control for content, context, and intent in com- munication. This results in unreliable models that lack basic knowledge, hallucinate facts, and dis- criminate users (Bender et al., 2021; Gehman et al., 2020; Pagnoni et al., 2021). Controlled text generation—sampling text from LMs to satisfy constraints on the properties of gen- erated text—aims to address these issues. Prior works incorporate constraints in existing decoding 1The code is available at: https://github.com/ Sachin19/mucoco/tree/sampling Figure 1: MUCOLA, our proposed method, stylized as µCOLA. Given a language model, a prompt/input x, and desired constraints defined as thresholds on dif- ferentiable functions, we perform Langevin Dynamics updates to generate the entire output sequence y non- autogressively. We show experiments highlighting both hard and soft constraints (§4). algorithms at token level by modifying output prob- abilities (Dathathri et al., 2020; Yang and Klein, 2021a; Krause et al., 2020a; Liu et al., 2021a; Lu et al., 2021b; Pascual et al., 2021; Liu et al., 2021b). While effective in certain settings, by generating au- toregressively (i.e., left-to-right), these approaches fail to account for global context and hardly gener- alize beyond a single constraint. More importantly, by modifying output probabilities, they end up alter- ing the underlying LM distribution compromising the fluency and task accuracy (Kumar et al., 2021). We propose an algorithm to sample text non- autoregressively from a conditional or uncondi- tional LM trained to perform any language gen- eration task—translation, summarization, dialog, prompt completion—while controlling for multi- ple, potentially competing constraints, and without sacrificing the base model quality. …

Content truncated to 500 words