Stochastic Wasserstein Autoencoder for Probabilistic Sentence Generation

Proceedings of NAACL-HLT 2019, pages 4068–4076 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4068 Stochastic Wasserstein Autoencoder for Probabilistic Sentence Generation Hareesh Bahuleyan,† Lili Mou,† Hao Zhou,‡ Olga Vechtomova† †University of Waterloo ‡ByteDance AI Lab hpallika@uwaterloo.ca doublepower.mou@gmail.com zhouhao.nlp@bytedance.com ovechtomova@uwaterloo.ca Abstract The variational autoencoder (VAE) imposes a probabilistic distribution (typically Gaus- sian) on the latent space and penalizes the Kullback–Leibler (KL) divergence between the posterior and prior. In NLP, VAEs are ex- tremely difficult to train due to the problem of KL collapsing to zero. One has to implement various heuristics such as KL weight anneal- ing and word dropout in a carefully engineered manner to successfully train a VAE for text. In this paper, we propose to use the Wasser- stein autoencoder (WAE) for probabilistic sen- tence generation, where the encoder could be either stochastic or deterministic. We show theoretically and empirically that, in the orig- inal WAE, the stochastically encoded Gaus- sian distribution tends to become a Dirac-delta function, and we propose a variant of WAE that encourages the stochasticity of the en- coder. Experimental results show that the la- tent space learned by WAE exhibits proper- ties of continuity and smoothness as in VAEs, while simultaneously achieving much higher BLEU scores for sentence reconstruction.1 1 Introduction Natural language sentence generation in the deep learning regime typically uses a recurrent neural network (RNN) to predict the most probable next word given previous words (Mikolov et al., 2010). Such RNN architecture can be further conditioned on some source information, for example, an in- put sentence, resulting in a sequence-to-sequence (Seq2Seq) model. Traditionally, sentence generation is accom- plished in a deterministic fashion, i.e., the model uses a deterministic neural network to encode an 1Our code is availabe at https://github.com/ HareeshBahuleyan/probabilistic_nlg A preliminary version of this paper was preprinted at https://arxiv.org/abs/1806.08462 input sentence to some hidden representations, from which it then decodes an output sentence us- ing another deterministic neural network. Bowman et al. (2016) propose to use the vari- ational autoencoder (VAE, Kingma and Welling, 2014) to map an input sentence to a probabilistic continuous latent space. VAE makes it possible to generate sentences from a distribution, which is desired in various applications. For example, in an open-domain dialog system, the informa- tion of an utterance and its response is not nec- essarily a one-to-one mapping, and multiple plau- sible responses could be suitable for a given input. Probabilistic sentence generation makes the dia- log system more diversified and more meaning- ful (Serban et al., 2017; Bahuleyan et al., 2018). Besides, probabilistic modeling of the hidden rep- resentations serves as a way of posterior regular- ization (Zhang et al., 2016), facilitating interpola- tion (Bowman et al., 2016) and manipulation of the latent representation (Hu et al., 2017). However, training VAEs in NLP is more diffi- cult than the image domain (Kingma and Welling, 2014). The VAE training involves a reconstruction loss and a Kullback–Leibler (KL) divergence be- tween the posterior and prior of the latent …

Content truncated to 500 words