Do sequence-to-sequence VAEs learn global features of sentences?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4296–4318, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4296 Do sequence-to-sequence VAEs learn global features of sentences? Tom Bosc Mila, Universit´e de Montr´eal bosct@mila.quebec Pascal Vincent Mila, Universit´e de Montr´eal, CIFAR vincentp@iro.umontreal.ca Abstract Autoregressive language models are powerful and relatively easy to train. However, these models are usually trained without explicit conditioning labels and do not offer easy ways to control global aspects such as sentiment or topic during generation. Bowman et al. (2016) adapted the Variational Autoencoder (VAE) for natural language with the sequence- to-sequence architecture and claimed that the latent vector was able to capture such global features in an unsupervised manner. We ques- tion this claim. We measure which words ben- efit most from the latent information by de- composing the reconstruction loss per position in the sentence. Using this method, we find that VAEs are prone to memorizing the first words and the sentence length, producing lo- cal features of limited usefulness. To alleviate this, we investigate alternative architectures based on bag-of-words assumptions and lan- guage model pretraining. These variants learn latent variables that are more global, i.e., more predictive of topic or sentiment labels. More- over, using reconstructions, we observe that they decrease memorization: the first word and the sentence length are not recovered as ac- curately than with the baselines, consequently yielding more diverse reconstructions. 1 Introduction The problem of generating natural language under- lies many classical NLP tasks such as translation, summarization, paraphrasing, etc. The problem is often formulated as learning a probabilistic model of sentences, then searching for probable sentences under this model. Expressive language models are typically built using neural networks (Bengio et al., 2003; Mikolov et al., 2010). Whether based on LSTMs (Hochreiter and Schmidhuber, 1997; Sundermeyer et al., 2012) or Transformers (Vaswani et al., 2017; Radford et al., 2019), language models are mostly autoregressive: the probability of a sentence is the product of the probability of each word given the previous words. By contrast, Bowman et al. (2016) built a Varia- tional Autoencoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) out of a sequence-to- sequence architecture (seq2seq) (Sutskever et al., 2014). It generates text in a two-step process: first, a latent vector is sampled from a prior distribu- tion; then, words are sampled from the probabil- ity distribution produced by the autoregressive de- coder, conditionally on the latent vector. The goal was to encourage a useful information decomposi- tion, where latent vectors would “explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features” (Bowman et al., 2016), while the more local correlations would be handled by the recurrent decoder. In principle, such a decomposition can be the basis for many applications. For example, using a single, unannotated corpus, it could enable para- phrasing (Roy and Grangier, 2019) or style transfer (Xu et al., 2019). For tasks requiring conditional generation such as machine translation or dialogue modeling, we could enforce …

Content truncated to 500 words