Generative Text Modeling through Short Run Inference

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1156–1165 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1156 Generative Text Modeling through Short Run Inference Bo Pang∗ UCLA bopang@ucla.edu Erik Nijkamp∗ UCLA enijkamp@ucla.edu Tian Han Stevens Institute of Technology than6@stevens.edu Ying Nian Wu UCLA ywu@stat.ucla.edu Abstract Latent variable models for text, when trained successfully, accurately model the data distri- bution and capture global semantic and syntac- tic features of sentences. The prominent ap- proach to train such models is variational au- toencoders (VAE). It is nevertheless challeng- ing to train and often results in a trivial local optimum where the latent variable is ignored and its posterior collapses into the prior, an is- sue known as posterior collapse. Various tech- niques have been proposed to mitigate this is- sue. Most of them focus on improving the in- ference model to yield latent codes of higher quality. The present work proposes a short run dynamics for inference. It is initialized from the prior distribution of the latent vari- able and then runs a small number (e.g., 20) of Langevin dynamics steps guided by its pos- terior distribution. The major advantage of our method is that it does not require a separate inference model or assume simple geometry of the posterior distribution, thus rendering an automatic, natural and flexible inference en- gine. We show that the models trained with short run dynamics more accurately model the data, compared to strong language model and VAE baselines, and exhibit no sign of posterior collapse. Analyses of the latent space show that interpolation in the latent space is able to generate coherent sentences with smooth transition and demonstrate improved classifi- cation over strong baselines with latent fea- tures from unsupervised pretraining. These re- sults together expose a well-structured latent space of our generative model. 1 Introduction The state-of-the-art language models (LM) are of- ten modeled with recurrent neural networks (RNN) (Mikolov et al., 2010) or attention-based models ∗Equal contributions. (Dong et al., 2019; Vaswani et al., 2017). They are optimized by making a series of next-step pre- dictions, encouraging the models to capture local dependency rather than global semantic features or high-level syntactic properties. A seminal work by Bowman et al. (2016) extends the standard LM to incorporate a continuous latent space which is aimed to explicitly capture global features. They formulate and train the model as a varational au- toencoder (VAE) (Kingma and Welling, 2014). In- deed, the model is able to generate coherent and di- verse sentences through continuous sampling, and provide smooth interpolation between sentences, uncovering a well-formed latent space. However, training VAE for text is challenging and often leads to a trivial local optimum, posterior col- lapse. Specifically, the training objective of VAE can be decomposed into a reconstruction term and a KL term that regularizes the distance between the posterior and prior of the latent variable. Due to the autoregressive nature of the decoder, it is able to reconstruct the …

Content truncated to 500 words