Generating Sentences from a Continuous Space

Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 10–21, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Generating Sentences from a Continuous Space Samuel R. Bowman∗ NLP Group and Dept. of Linguistics Stanford University sbowman@stanford.edu Luke Vilnis∗ CICS UMass Amherst luke@cs.umass.edu Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz & Samy Bengio Google Brain Google, Inc. {vinyals, adai, bengio}@google.com, rafjoz@gmail.com Abstract The standard recurrent neural network language model (rnnlm) generates sen- tences one word at a time and does not work from an explicit global sentence rep- resentation. In this work, we introduce and study an rnn-based variational au- toencoder generative model that incorpo- rates distributed latent representations of entire sentences. This factorization al- lows it to explicitly model holistic prop- erties of sentences such as style, topic, and high-level syntactic features. Samples from the prior over these sentence repre- sentations remarkably produce diverse and well-formed sentences through simple de- terministic decoding. By examining paths through this latent space, we are able to generate coherent novel sentences that in- terpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model’s latent sentence space, and present negative results on the use of the model in language modeling. 1 Introduction Recurrent neural network language models (rnnlms, Mikolov et al., 2011) represent the state of the art in unsupervised generative modeling for natural language sentences. In supervised settings, rnnlm decoders conditioned on task- specific features are the state of the art in tasks like machine translation (Sutskever et al., 2014; Bahdanau et al., 2015) and image captioning (Vinyals et al., 2015; Mao et al., 2015; Donahue et al., 2015). The rnnlm generates sentences word-by-word based on an evolving distributed state representation, which makes it a proba- bilistic model with no significant independence ∗First two authors contributed equally. Work was done when all authors were at Google, Inc. i went to the store to buy some groceries . i store to buy some groceries . i were to buy any groceries . horses are to buy any groceries . horses are to buy any animal . horses the favorite any animal . horses the favorite favorite animal . horses are my favorite animal . Table 1: Sentences produced by greedily decoding from points between two sentence encodings with a conventional autoencoder. The intermediate sen- tences are not plausible English. assumptions, and makes it capable of modeling complex distributions over sequences, including those with long-term dependencies. However, by breaking the model structure down into a series of next-step predictions, the rnnlm does not expose an interpretable representation of global features like topic or of high-level syntactic properties. We propose an extension of the rnnlm that is designed to explicitly capture such global features in a continuous latent variable. Naively, maxi- mum likelihood learning in such a model presents an intractable inference problem. Drawing inspi- ration from recent successes in …

Content truncated to 500 words