Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2079–2089 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2079 Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models Dinghan Shen1⇤, Asli Celikyilmaz2, Yizhe Zhang2, Liqun Chen1, Xin Wang3, Jianfeng Gao2, Lawrence Carin1 1 Duke University 2 Microsoft Research, Redmond 3 University of California, Santa Barbara dinghan.shen@duke.edu Abstract Variational autoencoders (VAEs) have re- ceived much attention recently as an end-to- end architecture for text generation with la- tent variables. However, previous works typ- ically focus on synthesizing relatively short sentences (up to 20 words), and the poste- rior collapse issue has been widely identified in text-VAEs. In this paper, we propose to leverage several multi-level structures to learn a VAE model for generating long, and coher- ent text. In particular, a hierarchy of stochas- tic layers between the encoder and decoder networks is employed to abstract more infor- mative and semantic-rich latent codes. Be- sides, we utilize a multi-level decoder struc- ture to capture the coherent long-term struc- ture inherent in long-form texts, by generating intermediate sentence representations as high- level plan vectors. Extensive experimental re- sults demonstrate that the proposed multi-level VAE model produces more coherent and less repetitive long text compared to baselines as well as can mitigate the posterior-collapse is- sue. 1 Introduction The variational autoencoder (VAE) for text (Bow- man et al., 2016) is a generative model in which a stochastic latent variable provides additional infor- mation to modulate the sequential text-generation process. VAEs have been used for various text processing tasks (Semeniuta et al., 2017; Zhao et al., 2017; Kim et al., 2018; Du et al., 2018; Hashimoto et al., 2018; Shen et al., 2018a; Xu and Durrett, 2018; Wang et al., 2019). While most recent work has focused on generating relatively short sequences (e.g., a single sentence or multi- ple sentences up to around twenty words), gen- erating long-form text (e.g., a single or multiple ⇤ This research was carried out during an internship at Microsoft Research. flat-VAE (standard) multilevel-VAE (our model) i went here for a grooming and a dog . it was very good . the owner is very nice and friendly . the owner is really nice and friendly . i don t know what they are doing . i have been going to this nail salon for over a year now . the last time i went there . the stylist was nice . but the lady who did my nails . she was very rude and did not have the best nail color i once had . the staff is very friendly and help- ful . the only reason i can t give them 5 stars . the only reason i am giving the ticket is because of the ticket . can t help but the staff is so friendly and helpful . can t help but the parking lot is just the same . i am a huge …

Content truncated to 500 words