Topic-Guided Variational Auto-Encoder for Text Generation

Proceedings of NAACL-HLT 2019, pages 166–177 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 166 Topic-Guided Variational Autoencoders for Text Generation Wenlin Wang1, Zhe Gan2, Hongteng Xu1,3, Ruiyi Zhang1, Guoyin Wang1, Dinghan Shen1, Changyou Chen4, Lawrence Carin1 1Duke University, 2Microsoft Dynamics 365 AI Research, 3Infinia ML, Inc, 4University at Buffalo {wenlin.wang, ruiyi.zhang, gw60, ds448, lcarin}@duke.edu zhe.gan@microsoft.com, hongteng.xu@infiniaml.com changyou@buffalo.edu Abstract We propose a topic-guided variational au- toencoder (TGVAE) model for text genera- tion. Distinct from existing variational au- toencoder (VAE) based approaches, which as- sume a simple Gaussian prior for the latent code, our model specifies the prior as a Gaus- sian mixture model (GMM) parametrized by a neural topic module. Each mixture com- ponent corresponds to a latent topic, which provides guidance to generate sentences un- der the topic. The neural topic module and the VAE-based neural sequence module in our model are learned jointly. In particular, a se- quence of invertible Householder transforma- tions is applied to endow the approximate pos- terior of the latent code with high flexibility during model inference. Experimental results show that our TGVAE outperforms alterna- tive approaches on both unconditional and con- ditional text generation, which can generate semantically-meaningful sentences with vari- ous topics. 1 Introduction Text generation plays an important role in vari- ous natural language processing (NLP) applica- tions, such as machine translation (Cho et al., 2014; Sutskever et al., 2014), dialogue generation (Li et al., 2017a), and text summarization (Nallapati et al., 2016; Rush et al., 2015). As a competitive solution to this task, the variational autoencoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) has been widely used in text-generation sys- tems (Bowman et al., 2015; Hu et al., 2017; Serban et al., 2017). In particular, VAE defines a gen- erative model that propagates latent codes drawn from a simple prior through a decoder to mani- fest data samples. The generative model is further augmented with an inference network, that feeds observed data samples through an encoder to yield a distribution on the corresponding latent codes. Compared with other potential methods, e.g., those based on generative adversarial networks (GANs) (Yu et al., 2017; Guo et al., 2017; Zhang et al., 2017b, 2018; Chen et al., 2018), VAE is of particular interest when one desires not only text generation, but also the capacity to infer meaning- ful latent codes from text. Ideally, semantically- meaningful latent codes can provide high-level guidance while generating sentences. For exam- ple, when generating text, the vocabulary could potentially be narrowed down if the input latent code corresponds to a certain topic (e.g., the word “military” is unlikely to appear in a sports-related document). However, in practice this desirable property is not fully achieved by existing VAE-based text gen- erative models, because of the following two chal- lenges. First, the sentences in documents may as- sociate with different semantic information (e.g., topic, sentiment, etc.) while the latent codes of ex- isting VAE-based text generative models often em- ploy a simple Gaussian …

Content truncated to 500 words