Continuous Language Generative Flow

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4609–4622 August 1–6, 2021. ©2021 Association for Computational Linguistics 4609 Continuous Language Generative Flow Zineng Tang Shiyue Zhang Hyounghun Kim Mohit Bansal UNC Chapel Hill {terran, shiyue, hyounghk, mbansal}@cs.unc.edu Abstract Recent years have witnessed various types of generative models for natural language gener- ation (NLG), especially RNNs or transformer based sequence-to-sequence models, as well as variational autoencoder (VAE) and gener- ative adversarial network (GAN) based mod- els. However, flow-based generative models, which achieve strong performance in image generation due to their invertibility and exact density estimation properties, have been less explored for NLG. In this paper, we propose a flow-based language generation model by adapting previous flow generative models to language generation via continuous input em- beddings, adapted affine coupling structures, and a novel architecture for autoregressive text generation. We also apply our framework to Sequence-to-Sequence generation, includ- ing text- and video-based Question Generation (QG) and Neural Machine Translation (NMT), and data augmentation for Question Answer- ing (QA). We use our language flow model to provide extra input features for QG and NMT, which achieves improvements over the strong QG baselines on SQuAD and TVQA and NMT baseline on WMT16. We also aug- ment QA data with new context by injecting noise to the latent features of the language flow and show this augmentation leads to a large performance improvement from strong base- lines on SQuAD and TVQA.1 1 Introduction Several generative models have been proposed for language generation, including sequence-to- sequence models based on RNNs (Luong et al., 2015) and transformers (Vaswani et al., 2017), as well as variational autoencoders (VAEs) to gen- erate diverse texts (Bowman et al., 2016; Jain 1Our code and models are available at: https:// github.com/zinengtang/ContinuousFlowNLG et al., 2017), plus generative adversarial networks (GANs) (Yu et al., 2017) to improve intended se- mantic fidelity. Another line of the generative model, normalizing flow (Rezende and Mohamed, 2015), is widely explored in computer vision and representation learning but less explored for NLG tasks. Flow models have been shown to be capable of improving probability density estimation, includ- ing variational inference (Rezende and Mohamed, 2015) and exact density estimation (Dinh et al., 2015). Generative flow is one type of flow model and first proposed by Dinh et al. (2015, 2017); Kingma and Dhariwal (2018). Taking advantage of its invertible structure, it can perform an exact density estimation of the input distribution. Thus, during generation, we can sample from its latent space and then generate new examples through its invertible decoder. Generative flow shows strong performance on image generation, attribute manip- ulation, and latent space inference (Kingma and Dhariwal, 2018). Considering these successful applications, we conjecture that the flow model should also have strong potential to be adapted for language generation tasks. Therefore, in this pa- per, we introduce a continuous language generative flow model that can deal with discrete language data in continuous latent space. …

Content truncated to 500 words