SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11575–11596 July 9-14, 2023 ©2023 Association for Computational Linguistics SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control Xiaochuang Han♠ Sachin Kumar♣ Yulia Tsvetkov♠ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ♣Language Technologies Institute, Carnegie Mellon University {xhan77, yuliats}@cs.washington.edu♠ sachink@cs.cmu.edu♣ Abstract Despite the growing success of diffusion mod- els in continuous-valued domains (e.g., im- ages), similar efforts for discrete domains such as text have yet to match the performance of au- toregressive language models. In this work, we present SSD-LM—a diffusion-based language model with two key design choices. First, SSD- LM is semi-autoregressive, iteratively gener- ating blocks of text, allowing for flexible out- put length at decoding time while enabling lo- cal bidirectional context updates. Second, it is simplex-based, performing diffusion on the natural vocabulary space rather than a learned latent space, allowing us to incorporate clas- sifier guidance and modular control using off- the-shelf classifiers without any adaptation. We evaluate SSD-LM on unconstrained text gen- eration benchmarks, and show that it matches or outperforms strong autoregressive GPT-2 models across standard quality and diversity metrics, while vastly outperforming diffusion- based baselines. On controlled text generation, SSD-LM also outperforms competitive base- lines, with an extra advantage in modularity.1 1 Introduction Diffusion models (Sohl-Dickstein et al., 2015), trained to iteratively refine noised inputs, have re- cently emerged as powerful tools for generative modeling in several continuous-valued domains such as images (Ho et al., 2020), audio (Kong et al., 2021), video (Ho et al., 2022), among others. At- tempts to adapt them for discrete domains such as text data, however, have only had limited success: prior work have shown to be promising on special- ized cases and small datasets (Hoogeboom et al., 2021; Austin et al., 2021; Li et al., 2022; Chen et al., 2022), but diffusion models for text still underper- form (and thus are not widely adopted) compared to autoregressive language models (AR-LMs) which 1Our code and models can be found at https:// github.com/xhan77/ssd-lm. remain the state-of-the-art general purpose text gen- erators (Radford et al., 2019; Brown et al., 2020). Despite potential advantages of diffusion models for text, there are two key challenges. First, dif- fusion models generate text non-autoregressively, i.e., they generate (and update) the entire sequence simultaneously rather than token by token left-to- right. Although this property is useful in practice since each output token is informed by a broader bi-directional context (Lee et al., 2018; Ghazvinine- jad et al., 2019), it requires pre-defining an output sequence length. This limits the flexibility and ap- plicability of trained models. On the other hand, non-autoregressive training with long sequences is expensive and difficult to optimize. In this work, we propose a semi-autoregressive solution which strikes a balance between length flexibility and the ability to alter previously generated tokens. A major advantage of diffusion models over the current standard of autoregressive LMs is their post-hoc controllability using …

Content truncated to 500 words