Long Text Generation with Topic-aware Discrete Latent Variable Model

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8100 - 8107 December 7-11, 2022 ©2022 Association for Computational Linguistics Long Text Generation with Topic-aware Discrete Latent Variable Model Erguang Yang1∗, Mingtong Liu2 , Deyi Xiong3, Yujie Zhang1†, Jinan Xu1 , Yufeng Chen1 1School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China 2Beijing Lanzhou Technology Co., Ltd., Beijing, China 3College of Intelligence and Computing, Tianjin University, Tianjin, China {egyang,yjzhang,jaxu,chenyf}@bjtu.edu.cn, liumingtong@langboat.com, dyxiong@tju.edu.cn Abstract Generating coherent long texts is an important yet challenging task, particularly for the open- ended generation task. Prior work based on discrete latent codes focuses on the modeling of discourse relation, resulting in discrete codes only learning shallow semantics (Ji and Huang, 2021). A natural text always revolves around several related topics and the transition across them is natural and smooth. In this work, we investigate whether discrete latent codes can learn information of topics. To this end, we build a topic-aware latent code-guided text gen- eration model. To encourage discrete codes to model information about topics, we propose a span-level bag-of-words training objective for the model. Automatic and manual evaluation experiments show that our method can generate more topic-relevant and coherent texts. 1 Introduction Generating coherent long texts based on a shorter text input is an important yet challenging task. Recent large-scale pre-trained language models (LMs), e.g. GPT-2 (Radford et al., 2019) and BART (Lewis et al., 2020) have shown state-of-the-art per- formance on various natural language generation (NLG) tasks such as summarization and dialog gen- eration (Radford et al., 2019; Lewis et al., 2020). Although pre-trained language models can gener- ate fluent texts, it is still challenging for them to generate coherent long-form texts. To address this, various text generation methods have been proposed to improve the coherence of generated texts. The first type of methods attempt to decompose the long text generation procedure into two stages: first producing a skeleton and then generating a text based on the skeleton, where the skeleton can be a set of keywords (Yao et al., 2019; Hua and Wang, 2020; Tan et al., 2021), events ∗This work was done when Erguang Yang was interning at Beijing Lanzhou Technology Co., Ltd., Beijing, China. †Corresponding Author. (Goldfarb-Tarrant et al., 2020; Li et al., 2022), etc. However, these methods more or less rely on high- precision automatic extraction tools. The second line of methods introduce latent vari- ables to model high-level structures of texts (Shao et al., 2019; Shen et al., 2019; Ji and Huang, 2021). Shen et al. (2019) develop multi-level VAE models (Bowman et al., 2016) for modeling paragraph- level and sentence-level features. Recently, Ji and Huang (2021) have explored discrete latent vari- ables (Van Den Oord et al., 2017) to model inter- sentence relations and achieved impressive perfor- mance. However, through preliminary experiments, we have found that their methods often produced texts unrelated to the input prompts, as shown in Table 5. The possible reason is that latent variables only learn shallow semantic …

Content truncated to 500 words