Variational Attention for Sequence-to-Sequence Models

Proceedings of the 27th International Conference on Computational Linguistics, pages 1672–1682 Santa Fe, New Mexico, USA, August 20-26, 2018. 1672 Variational Attention for Sequence-to-Sequence Models Hareesh Bahuleyan∗† Lili Mou∗‡ Olga Vechtomova† Pascal Poupart† †University of Waterloo, Canada {hpallika, ovechtomova, ppoupart}@uwaterloo.ca ‡AdeptMind Research, Toronto, Canada doublepower.mou@gmail.com Abstract The variational encoder-decoder (VED) encodes source information as a set of random variables using a neural network, which in turn is decoded into target data using another neural network. In natural language processing, sequence-to-sequence (Seq2Seq) models typically serve as encoder- decoder networks. When combined with a traditional (deterministic) attention mechanism, the variational latent space may be bypassed by the attention model, and thus becomes ineffective. In this paper, we propose a variational attention mechanism for VED, where the attention vector is also modeled as Gaussian distributed random variables. Results on two experiments show that, without loss of quality, our proposed method alleviates the bypassing phenomenon as it increases the diversity of generated sentences.1 1 Introduction The variational autoencoder (VAE), proposed by Kingma and Welling (2014), encodes data to latent (ran- dom) variables, and then decodes the latent variables to reconstruct the input data. Theoretically, it opti- mizes a variational lower bound of the log-likelihood of the data. Compared with traditional variational methods such as mean-field approximation (Wainwright et al., 2008), VAE leverages modern neural net- works and hence is a more powerful density estimator. Compared with traditional autoencoders (Hinton and Salakhutdinov, 2006), which are deterministic, VAE populates hidden representations to a region (in- stead of a single point), making it possible to generate diversified data from the vector space (Bowman et al., 2016) or even control the generated samples (Hu et al., 2017). In natural language processing (NLP), recurrent neural networks (RNNs) are typically used as both the encoder and decoder, known as a sequence-to-sequence (Seq2Seq) model. Although variational Seq2Seq models are much trickier to train in comparison to the image domain, Bowman et al. (2016) succeed in training a sequence-to-sequence VAE and generating sentences from a continuous latent space. Such an architecture can further be extended to a variational encoder-decoder (VED) to transform one sequence into another with the “variational” property (Serban et al., 2017; Zhou and Neubig, 2017). When applying attention mechanisms (Bahdanau et al., 2015) to variational Seq2Seq models, however, we find the generated sentences are of less variety, implying that the variational latent space is ineffec- tive. The attention mechanism summarizes source information as an attention vector by weighted sum, where the weights are a learned probabilistic distribution; then the attention vector is fed to the decoder. Evidence shows that attention significantly improves Seq2Seq performance in translation (Bahdanau et al., 2015), summarization (Rush et al., 2015), etc. In variational Seq2Seq, however, the attention mecha- nism unfortunately serves as a “bypassing” mechanism. In other words, the variational latent space does not need to learn much, as long as the attention mechanism itself is powerful enough to capture source information. In this paper, we propose a variational attention mechanism to address this …

Content truncated to 500 words