Progressive Generation of Long Text with Pretrained Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4313–4324 June 6–11, 2021. ©2021 Association for Computational Linguistics 4313 Progressive Generation of Long Text with Pretrained Language Models Bowen Tan1, Zichao Yang1, Maruan Al-Shedivat1, Eric P. Xing1,2,3, Zhiting Hu1,4 1Carnegie Mellon University, 2Petuum Inc., 3MBZUAI, 4UC San Diego {btan2, zichaoy, alshedivat, epxing}@andrew.cmu.edu, zhh019@ucsd.edu Abstract Large-scale language models (LMs) pre- trained on massive corpora of text, such as GPT-2, are powerful open-domain text genera- tors. However, as our systematic examination reveals, it is still challenging for such models to generate coherent long passages of text (e.g., 1000 tokens), especially when the models are fine-tuned to the target domain on a small cor- pus. Previous planning-then-generation meth- ods also fall short of producing such long text in various domains. To overcome the limitations, we propose a simple but effec- tive method of generating text in a progressive manner, inspired by generating images from low to high resolution. Our method first pro- duces domain-specific content keywords and then progressively refines them into complete passages in multiple stages. The simple de- sign allows our approach to take advantage of pretrained LMs at each stage and effectively adapt to any target domain given only a small set of examples. We conduct a comprehensive empirical study with a broad set of evaluation metrics, and show that our approach signifi- cantly improves upon the fine-tuned large LMs and various planning-then-generation methods in terms of quality and sample efficiency. Hu- man evaluation also validates that our model generations are more coherent.1 1 Introduction Generating coherent long text (e.g., 1000s of to- kens) is useful in myriad applications of creating re- ports, essays, and other long-form content. Yet the problem is particularly challenging as it demands models to capture global context, plan content, and produce local words in a consistent manner. Prior studies on “long” text generation have typically limited to outputs of 50-200 tokens (Shen et al., 2019; Bosselut et al., 2018; Zhao et al., 2020). 1Code available at https://github.com/ tanyuqian/progressive-generation Figure 1: Results of large-scale LMs (GPT-2 and BART) fine-tuned on 10K stories. Coherence of text is evaluated by BERT next sentence prediction (NSP) score, where x-axis is the position of the evaluated sentences in the passage. There is a significant gap in coherence between text by human and text by large-scale LMs. Our proposed ProGen instead generates more coherent samples close to human text. Recent large-scale pretrained language models (LMs), such as GPT-2 (Radford et al., 2019) and BART (Lewis et al., 2020), emerged as an impres- sive open-ended text generator capable of produc- ing surprisingly fluent text. The massive LMs are typically pretrained on large corpora of generic text once, and then fine-tuned with small domain- specific data. The latest work has mostly focused on the regime of relatively short text with low hun- dreds of tokens. For example, Holtzman et al. (2020); See et al. (2019); Hua and Wang (2020) studied GPT-2 and …

Content truncated to 500 words