Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling

Proceedings of NAACL-HLT 2019, pages 284–294 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 284 Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling Prince Zizhuang Wang Department of Computer Science University of California, Santa Barbara zizhuang wang@ucsb.edu William Yang Wang Department of Computer Science University of California, Santa Barbara william@cs.ucsb.edu Abstract Recurrent Variational Autoencoder has been widely used for language modeling and text generation tasks. These models often face a difficult optimization problem, also known as the Kullback-Leibler (KL) term vanishing is- sue, where the posterior easily collapses to the prior, and the model will ignore latent codes in generative tasks. To address this problem, we introduce an improved Wasserstein Varia- tional Autoencoder (WAE) with Riemannian Normalizing Flow (RNF) for text modeling. The RNF transforms a latent variable into a space that respects the geometric character- istics of input space, which makes posterior impossible to collapse to the non-informative prior. The Wasserstein objective minimizes the distance between the marginal distribution and the prior directly, and therefore does not force the posterior to match the prior. Empir- ical experiments show that our model avoids KL vanishing over a range of datasets and has better performances in tasks such as language modeling, likelihood approximation, and text generation. Through a series of experiments and analysis over latent space, we show that our model learns latent distributions that re- spect latent space geometry and is able to gen- erate sentences that are more diverse. 1 1 Introduction Variational Autocoder (VAE) (Kingma and Welling, 2013; Rezende and Mohamed, 2015) is a probabilistic generative model shown to be successful over a wide range of tasks such as image generation (Gregor et al., 2015; Yan et al., 2016), dialogue generation (Zhao et al., 2017b), transfer learning (Shen et al., 2017), and classification (Jang et al., 2017). The encoder- decoder architecture of VAE allows it to learn a 1Code could be found at https://github.com/ kingofspace0wzz/wae-rnf-lm continuous space of latent representations from high-dimensional data input and makes sampling procedure from such latent space very straightfor- ward. Recent studies also show that VAE learns meaningful representations that encode non-trivial information from input (Gao et al., 2018; Zhao et al., 2017a). Applications of VAE in tasks of Natural Lan- guage Processing (Bowman et al., 2015; Zhao et al., 2017b, 2018; Miao et al., 2016) is not as suc- cessful as those in Computer Vision. With long- short-term-memory network (LSTM) (Hochre- iter and Schmidhuber, 1997) used as encoder- decoder model, the recurrent variational autoen- coder (Bowman et al., 2015) is the first ap- proach that applies VAE to language modeling tasks. They observe that LSTM decoder in VAE often generates texts without making use of la- tent representations, rendering the learned codes as useless. This phenomenon is caused by an op- timization problem called KL-divergence vanish- ing when training VAE for text data, where the KL-divergence term in VAE objective collapses to zero. This makes the learned representations meaningless as zero KL-divergence indicates that the …

Content truncated to 500 words