A Hybrid Convolutional Variational Autoencoder for Text Generation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 627–637 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A Hybrid Convolutional Variational Autoencoder for Text Generation Stanislau Semeniuta1 Aliaksei Severyn2 Erhardt Barth1 1Universit¨at zu L¨ubeck, Institut f¨ur Neuro- und Bioinformatik {stas,barth}@inb.uni-luebeck.de 2Google Research Europe severyn@google.com Abstract In this paper we explore the effect of archi- tectural choices on learning a variational autoencoder (VAE) for text generation. In contrast to the previously introduced VAE model for text where both the encoder and decoder are RNNs, we propose a novel hybrid architecture that blends fully feed-forward convolutional and deconvo- lutional components with a recurrent lan- guage model. Our architecture exhibits several attractive properties such as faster run time and convergence, ability to bet- ter handle long sequences and, more im- portantly, it helps to avoid the issue of the VAE collapsing to a deterministic model. 1 Introduction Generative models of texts are currently at the cornerstone of natural language understanding en- abling recent breakthroughs in machine transla- tion (Bahdanau et al., 2014; Wu et al., 2016), dia- logue modelling (Serban et al., 2016), abstractive summarization (Rush et al., 2015), etc. Currently, RNN-based generative models hold state-of-the-art results in both unconditional (J´ozefowicz et al., 2016; Ha et al., 2016) and con- ditional (Vinyals et al., 2014) text generation. At a high level, these models represent a class of au- toregressive models that work by generating out- puts sequentially one step at a time where the next predicted element is conditioned on the history of elements generated thus far. Variational autoencoders (VAE), recently intro- duced by (Kingma and Welling, 2013; Rezende et al., 2014), offer a different approach to genera- tive modeling by integrating stochastic latent vari- ables into the conventional autoencoder architec- ture. The primary purpose of learning VAE-based generative models is to be able to generate realis- tic examples as if they were drawn from the input data distribution by simply feeding noise vectors through the decoder. Additionally, the latent rep- resentations obtained by applying the encoder to input examples give a fine-grained control over the generation process that is harder to achieve with more conventional autoregressive models. Similar to compelling examples from image generation, where it is possible to condition generated human faces on various attributes such as hair, skin color and style (Yan et al., 2015; Larsen et al., 2015), in text generation it should be possible to also control various attributes of the generated sentences, such as, for example, sentiment or writing style. While training VAE-based models seems to pose little difficulty when applied to the tasks of generating natural images (Bachman, 2016; Gul- rajani et al., 2016) and speech (Fraccaro et al., 2016), their application to natural text generation requires additional care (Bowman et al., 2016; Miao et al., 2015). As discussed by Bowman et al. (2016), the core difficulty of training VAE models is the collapse of the latent loss (represented by the KL divergence term) to zero. In this case …

Content truncated to 500 words