Neural Gaussian Copula for Variational Autoencoder

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4333–4343, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4333 Neural Gaussian Copula for Variational Autoencoder Prince Zizhuang Wang Department of Computer Science University of California Santa Barbara zizhuang wang@ucsb.edu William Yang Wang Department of Computer Science University of California Santa Barbara william@cs.ucsb.edu Abstract Variational language models seek to estimate the posterior of latent variables with an ap- proximated variational posterior. The model often assumes the variational posterior to be factorized even when the true posterior is not. The learned variational posterior under this assumption does not capture the dependency relationships over latent variables. We ar- gue that this would cause a typical training problem called posterior collapse observed in all other variational language models. We propose Gaussian Copula Variational Autoen- coder (VAE) to avert this problem. Copula is widely used to model correlation and depen- dencies of high-dimensional random variables, and therefore it is helpful to maintain the de- pendency relationships that are lost in VAE. The empirical results show that by modeling the correlation of latent variables explicitly us- ing a neural parametric copula, we can avert this training difficulty while getting competi- tive results among all other VAE approaches. 1 1 Introduction Variational Inference (VI) (Wainwright et al., 2008; Hoffman et al., 2013) methods are inspired by calculus of variation (Gelfand et al., 2000). It can be dated back to the 18th century when it was mainly used to study the problem of the change of functional, which is defined as the mapping from functions to real space and can be understood as the function of functions. VI takes a distri- bution as a functional and then studies the prob- lem of matching this distribution to a target dis- tribution using calculus of variation. After the rise of deep learning (Krizhevsky et al., 2012), a deep generative model called Variational Au- toencoder (Kingma and Welling, 2014; Hoffman 1Code will be released at https://github.com/ kingofspace0wzz/copula-vae-lm Figure 1: Intuitive illustration of VI: The elliptic P is a distribution family containing the true posterior p ∈ P, and the circle Q is a Mean-field variational family con- taining a standard normal prior N. The optimal solu- tion q∗ is the one in Q that has the smallest KL(q||p). In reality these two families may not overlap. et al., 2013) is proposed based on the theory of VI and achieves great success over a huge number of tasks, such as transfer learning (Shen et al., 2017), unsupervised learning (Jang et al., 2017), image generation (Gregor et al., 2015), semi-supervised classification (Jang et al., 2017), and dialogue gen- eration (Zhao et al., 2017). VAE is able to learn a continuous space of latent random variables which are useful for a lot of classification and generation tasks. Recent studies (Bowman et al., 2015; Yang et al., 2017; Xiao et al., 2018; Xu and Durrett, 2018) show that …

Content truncated to 500 words