Implicit Deep Latent Variable Models for Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3946–3956, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3946 Implicit Deep Latent Variable Models for Text Generation Le Fang†, Chunyuan Li§, Jianfeng Gao§, Wen Dong†, Changyou Chen† †University at Buffalo, §Microsoft Research, Redmond {lefang, wendong, changyou}@buffalo.edu {chunyl, jfgao}@microsoft.com Abstract Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation. One key factor is the exploitation of smooth latent structures to guide the generation. How- ever, the representation power of VAEs is lim- ited due to two reasons: (1) the Gaussian as- sumption is often made on the variational pos- teriors; and meanwhile (2) a notorious “poste- rior collapse” issue occurs. In this paper, we advocate sample-based representations of vari- ational distributions for natural language, lead- ing to implicit latent features, which can pro- vide flexible representation power compared with Gaussian-based posteriors. We further de- velop an LVM to directly match the aggregated posterior to the prior. It can be viewed as a nat- ural extension of VAEs with a regularization of maximizing mutual information, mitigating the “posterior collapse” issue. We demonstrate the effectiveness and versatility of our models in various text generation scenarios, including language modeling, unaligned style transfer, and dialog response generation. The source code to reproduce our experimental results is available on GitHub1. 1 Introduction Deep latent variable models (LVM) such as varia- tional auto-encoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) are successfully ap- plied for many natural language processing tasks, including language modeling (Bowman et al., 2015; Miao et al., 2016), dialogue response generation (Zhao et al., 2017b), controllable text generation (Hu et al., 2017) and neural machine translation (Shah and Barber, 2018) etc. One advantage of VAEs is the flexible distribution-based latent rep- resentation. It captures holistic properties of in- put, such as style, topic, and high-level linguis- 1https://github.com/fangleai/ Implicit-LVM tic/semantic features, which further guide the gen- eration of diverse and relevant sentences. However, the representation capacity of VAEs is restrictive due to two reasons. The first reason is rooted in the assumption of variational posteri- ors, which usually follow spherical Gaussian dis- tributions with diagonal co-variance matrices. It has been shown that an approximation gap gener- ally exists between the true posterior and the best possible variational posterior in a restricted fam- ily (Cremer et al., 2018). Consequently, the gap may militate against learning an optimal genera- tive model, as its parameters may be always up- dated based on sub-optimal posteriors (Kim et al., 2018). The second reason is the so-called poste- rior collapse issue, which occurs when learning VAEs with an auto-regressive decoder (Bowman et al., 2015). It produces undesirable outcomes: the encoder yields meaningless posteriors that are very close to the prior, while the decoder tends to ignore the latent codes in generation (Bowman et al., 2015). Several attempts have been made to alleviate …

Content truncated to 500 words