DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4521–4534 July 9-14, 2023 ©2023 Association for Computational Linguistics DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models Zhengfu He∗ Tianxiang Sun∗ Qiong Tang Kuanning Wang Xuanjing Huang Xipeng Qiu† School of Computer Science, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {zfhe19,txsun19,wangkn20,xjhuang,xpqiu}@fudan.edu.cn qtang22@m.fudan.edu.cn Abstract We present DiffusionBERT, a new generative masked language model based on discrete dif- fusion models. Diffusion models and many pre- trained language models have a shared training objective, i.e., denoising, making it possible to combine the two powerful models and enjoy the best of both worlds. On the one hand, dif- fusion models offer a promising training strat- egy that helps improve the generation quality. On the other hand, pre-trained denoising lan- guage models (e.g., BERT) can be used as a good initialization that accelerates convergence. We explore training BERT to learn the reverse process of a discrete diffusion process with an absorbing state and elucidate several designs to improve it. First, we propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step based on the information of each token. Sec- ond, we investigate several designs of incorpo- rating the time step into BERT. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improve- ment over existing diffusion models for text (e.g., D3PM and Diffusion-LM) and previous generative masked language models in terms of perplexity and BLEU score. Promising re- sults in conditional generation tasks show that DiffusionBERT can generate texts of compa- rable quality and more diverse than a series of established baselines. 1 Introduction Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) have recently emerged as a new class of state-of-the-art gener- ative models, achieving high-quality synthesis re- sults on image data (Ramesh et al., 2022; Rom- bach et al., 2022; Saharia et al., 2022). Though these models captured widespread attention from ∗ Equal contribution. † Corresponding author. xT · · · xt xt−1 · · · x0 "[MASK] [MASK] [MASK]" "[MASK] world !" "Hello world !" pθ(xt−1|xt, t) q(xt|xt−1) (a) Diffusion models for discrete data xT · · · xt xt−1 · · · x0 "[MASK] [MASK] [MASK]" "Hello [MASK] !" "Hello world !" pθ(xt−1|xt) q(xt|xt−1, x0) (b) Non-Markovian DiffusionBERT Figure 1: In contrast to conventional discrete diffusion models, DiffusionBERT uses BERT as its backbone to perform text generation. The main differences are high- lighted in color: (1) DiffusionBERT performs decoding without knowing the current time step while canonical diffusion models are conditioned on time step. (2) The diffusion process of DiffusionBERT is non-Markovian in that it generates noise samples xt conditioning not only on xt−1 but also on x0. Such a non-Markov pro- cess is due to our proposed noise schedule. not only the research community but also the pub- lic, applying diffusion models to text data is still challenging and under-explored due to …

Content truncated to 500 words