A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3603–3614, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3603 A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text Bohan Li∗1, Junxian He∗1, Graham Neubig1, Taylor Berg-Kirkpatrick2, Yiming Yang1 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science and Engineering, University of California San Diego {bohanl1,junxianh,gneubig,yiming}@cs.cmu.edu, tberg@eng.ucsd.edu Abstract When trained effectively, the Variational Au- toencoder (VAE) is both a powerful language model and an effective representation learn- ing framework. In practice, however, VAEs are trained with the evidence lower bound (ELBO) as a surrogate objective to the in- tractable marginal data likelihood. This ap- proach to training yields unstable results, fre- quently leading to a disastrous local optimum known as posterior collapse. In this paper, we investigate a simple fix for posterior col- lapse which yields surprisingly effective re- sults. The combination of two known heuris- tics, previously considered only in isolation, substantially improves held-out likelihood, re- construction, and latent representation learn- ing when compared with previous state-of-the- art methods. More interestingly, while our experiments demonstrate superiority on these principle evaluations, our method obtains a worse ELBO. We use these results to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling.1 1 Introduction Latent variable models attempt to model observed data x given latent variables z, both for pur- poses of modeling data distributions p(x) (e.g. language modeling) and learning representations z for a particular x (e.g. sentence embedding). Variational Autoencoders (VAEs) (Kingma and Welling, 2014) are a powerful framework for learning latent variable models using neural net- works. The generative model of VAEs first sam- ples a latent vector z from a prior p(z), then ap- plies a neural decoder p(x|z) to produce x con- ditioned on the latent code z. VAEs are trained ∗Equal contribution. 1Code is available at https://github.com/ bohanli/vae-pretraining-encoder. to maximize the evidence lower bound (ELBO) of the intractable log marginal likelihood: Ez∼qφ(z|x)[log pθ(x|z)] − DKL(qφ(z|x)∥p(z)), where qφ(z|x) represents an approximate poste- rior distribution (i.e. the encoder or inference net- work) and pθ(x|z) is the generative distribution (i.e. the decoder). However, modeling text with VAEs has proven to be challenging, and is an open research prob- lem (Yang et al., 2017; Xu and Durrett, 2018; Kim et al., 2018; Dieng et al., 2018; He et al., 2019; Pelsmaeker and Aziz, 2019). When a strong de- coder (e.g. the LSTM (Hochreiter and Schmidhu- ber, 1997)) is employed, training often falls into a trivial local optimum where the decoder learns to ignore the latent variable and the encoder fails to encode any information. This phenomenon is referred to as “posterior collapse” (Bowman et al., 2016). Existing efforts tackling this prob- lem include re-weighting the KL loss (Bowman et al., 2016; Kingma et al., 2016; Liu et al., 2019), changing the model (Yang …

Content truncated to 500 words