Better Exploiting Latent Variables in Text Modeling

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5527–5532 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5527 Better Exploiting Latent Variables in Text Modeling Canasai Kruengkrai Yahoo Japan Corporation ckruengk@yahoo-corp.jp Abstract We show that sampling latent variables multi- ple times at a gradient step helps in improving a variational autoencoder and propose a simple and effective method to better exploit these la- tent variables through hidden state averaging. Consistent gains in performance on two differ- ent datasets, Penn Treebank and Yahoo, indi- cate the generalizability of our method.1 1 Introduction Introducing latent variables to neural language models would help in generating plausible sen- tences that reflect sentential semantics (Bowman et al., 2016). The success of learning latent vari- ables is also beneficial to various natural lan- guage processing (NLP) tasks such as sentence compression (Miao and Blunsom, 2016) and text style transfer (Shen et al., 2017). One of the widely-used latent variable models is the varia- tional autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014). When applying the VAE to text data, recurrent neural networks are typically utilized for both the encoder and the de- coder. Training the VAE with a high-capacity de- coder such as a long short-term memory (LSTM) network (Hochreiter and Schmidhuber, 1997) can be challenging. The LSTM is powerful enough to model the underlying data distribution without the use of latent variables. In this paper, we take a closer look at one of the components in an LSTM-VAE model, namely the latent variable sampling scheme. Fig. 1 illustrates our baseline LSTM-VAE model built upon Bow- man et al. (2016)’s model. At each gradient step (i.e., a minibatch run), most previous work pairs an input sentence with a single latent variable de- noted by z. This would be sufficient in some 1The code for reproducibility is available at https:// research-lab.yahoo.co.jp/en/software. Word embedding Encoding LSTM cell Decoding LSTM cell Description Linear Linear+Softmax Dropout new hope a !"# a new hope a new hope $"# %∼'(0,1) ( ) * ⨁ ⨀ Figure 1: Baseline LSTM-VAE model. tasks but not necessarily effective in text model- ing. At the beginning of training, the latent vari- able z contains a small amount of information about the input sentence. Many latent units of z are pulled towards the prior early to optimize an objective function before they capture useful in- formation (Hoffman et al., 2013; Sønderby et al., 2016). Without a cost annealing strategy or a con- straint on the decoder (Bowman et al., 2016; Chen et al., 2017; Yang et al., 2017), z would be en- tirely ignored for the remaining training steps. In our work, we aim at developing a simple variant of the LSTM-VAE model to address this common training issue. We observe that pairing the input sentence with multiple latent variables improves latent variable usage. In addition, we present a method that leverages multiple latent variables to further boost the performance of the baseline LSTM-VAE model. …

Content truncated to 500 words