Enhancing Variational Autoencoders with Mutual Information Neural Estimation for Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4047–4057, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4047 Enhancing Variational Autoencoders with Mutual Information Neural Estimation for Text Generation Dong Qian, William K. Cheung Department of Computer Science Hong Kong Baptist University {dongqian,william}@comp.hkbu.edu.hk Abstract While broadly applicable to many natural lan- guage processing (NLP) tasks, variational au- toencoders (VAEs) are hard to train due to the posterior collapse issue where the latent vari- able fails to encode the input data effectively. Various approaches have been proposed to al- leviate this problem to improve the capability of the VAE. In this paper, we propose to in- troduce a mutual information (MI) term be- tween the input and its latent variable to regu- larize the objective of the VAE. Since estimat- ing the MI in the high-dimensional space is in- tractable, we employ neural networks for the estimation of the MI and provide a training al- gorithm based on the convex duality approach. Our experimental results on three benchmark datasets demonstrate that the proposed model, compared to the state-of-the-art baselines, ex- hibits less posterior collapse and has compara- ble or better performance in language model- ing and text generation. We also qualitatively evaluate the inferred latent space and show that the proposed model can generate more reason- able and diverse sentences via linear interpola- tion in the latent space. 1 Introduction Deep learning architectures are parameterized by families of non-linear functions, which learn mul- tiple levels of more abstract representations (Ben- gio, 2009; Bengio et al., 2013). Recently, there has been a surge of interest in deep generative models for unsupervised learning, such as variational au- toencoders (VAEs) (Kingma and Welling, 2014; Rezende et al., 2014), generative adversarial net- works (GANs) (Goodfellow et al., 2014), and au- toregressive models (van den Oord et al., 2016). The goal is to learn a compact representation to capture the salient structure in a given highly com- plex high-dimensional unlabelled data so that new data with some variations can be generated. They have been widely applied to a range of NLP tasks, such as language modeling (Bowman et al., 2016; Zhao et al., 2018a), dialog generation (Zhao et al., 2017, 2018b), etc. In this paper, we focus on the VAE with recurrent neural networks as its encoder and decoder for text generation. Recurrent neural networks (RNNs) are one of the state-of-the-art autoregressive models for text modeling (Melis et al., 2018; Trinh et al., 2018). Training RNNs involves factorizing the joint dis- tribution over a set of random variables into a se- ries of conditional distributions. Yet, the one-step- ahead predictions force RNNs to learn local cor- relations, rather than global coherence. This is in- sufficient to capture high-level abstractions which characterize text sequences. One approach to circumvent this challenge is to introduce variational autoencoders (VAEs) (Bow- man et al., 2016), where an encoder learns a latent variable from …

Content truncated to 500 words