Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 697 - 716 July 10-15, 2022 ©2022 Association for Computational Linguistics Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation Jinyi Hu1,2,3, Xiaoyuan Yi5, Wenhao Li1,2,3, Maosong Sun1,2,3,4∗, Xing Xie5 1 Department of Computer Science and Technology, Tsinghua University, Beijing, China 2 Beijing National Research Center for Information Science and Technology 3 Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4 International Innovation Center of Tsinghua University, Shanghai, China 5 Microsoft Research Asia hu-jy21@mails.tsinghua.edu.cn, xiaoyuanyi@microsoft.com Abstract The past several years have witnessed Vari- ational Auto-Encoder’s superiority in various text generation tasks. However, due to the se- quential nature of the text, auto-regressive de- coders tend to ignore latent variables and then reduce to simple language models, known as the KL vanishing problem, which would fur- ther deteriorate when VAE is combined with Transformer-based structures. To ameliorate this problem, we propose DELLA, a novel variational Transformer framework. DELLA learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product. In this way, DELLA forces these posterior latent variables to be fused deeply with the whole computation path and hence incorporate more information. We theoretically demonstrate that our method can be regarded as entangling latent variables to avoid posterior information decrease through layers, enabling DELLA to get higher non- zero KL values even without any annealing or thresholding tricks. Experiments on four un- conditional and three conditional generation tasks show that DELLA could better alleviate KL vanishing and improve both quality and di- versity compared to several strong baselines. 1 Introduction Variational Autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) has proven to be successful in generating various kinds of text, such as stylistic text (Hu et al., 2017; John et al., 2019), dialogue (Zhao et al., 2017), story (Yu et al., 2020) and poetry (Yi et al., 2020). The sequen- tial nature of the text leads to typically used auto- regressive decoders in VAE for language genera- tion. However, such strong decoders tend to evade the difficulty of learning meaningful latent codes by heavily relying on previously generated words ∗Corresponding author. Email: sms@tsinghua.edu.cn Embedding Layer Transformer Decoder Layer 1 Transformer Decoder Layer 2 ⋮ ⋮ Transformer Decoder Layer L Softmax Layer (c) x5 x1 x2 x3 x4 𝑧 (b) 𝑧 (a) 𝑧 Figure 1: Existing paradigms of Transformer VAE. and hence ignore latent variables (Bowman et al., 2016), known as KL vanishing or posterior col- lapse. This problem causes two drawbacks: (a) the posterior distribution quickly turns into the prior one (usually standard Gaussian), falling to build expressive latent representations; (b) the decoder reduces to a naive language model, resulting in monotonous generated text (Fu et al., 2019). To ameliorate this problem, researchers have de- signed various techniques. Among them, three broadly used methods include weakening de- coders …

Content truncated to 500 words