PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2288 - 2305 May 22-27, 2022 c⃝2022 Association for Computational Linguistics PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation Zhe Hu1, Hou Pong Chan2, Jiachen Liu1, Xinyan Xiao1, Hua Wu1, and Lifu Huang3 1Baidu Inc 2Faculty of Science and Technology, University of Macau 3Virginia Tech 1{huzhe01,liujiachen,xiaoxinyan,wu_hua}@baidu.com 2hpchan@um.edu.mo, 3lifuh@vt.edu Abstract Despite recent progress of pre-trained language models on generating fluent text, existing meth- ods still suffer from incoherence problems in long-form text generation tasks that require proper content control and planning to form a coherent high-level logical flow. In this work, we propose PLANET, a novel generation frame- work leveraging autoregressive self-attention mechanism to conduct content planning and surface realization dynamically. To guide the generation of output sentences, our framework enriches the Transformer decoder with latent representations to maintain sentence-level se- mantic plans grounded by bag-of-words. More- over, we introduce a new coherence-based con- trastive learning objective to further improve the coherence of output. Extensive experi- ments are conducted on two challenging long- form text generation tasks including counter- argument generation and opinion article gener- ation. Both automatic and human evaluations show that our method significantly outperforms strong baselines and generates more coherent texts with richer contents. 1 Introduction Neural sequence-to-sequence (seq2seq) models are dominant methods for text generation nowadays, which are trained to maximize the log-likelihood over targets in an end-to-end fashion (Cho et al., 2014). Recently, pre-trained methods such as GPT- 2 (Radford et al., 2019) and BART (Lewis et al., 2020) have achieved promising results by lever- aging large-scale data. While these models can generate fluent results, they still fall short of pro- ducing coherent long-form texts with multiple sen- tences (Dou et al., 2021). Long text generation, especially opinion gener- ation, usually requires the model to (1) conduct proper content selection and ordering (i.e., “what to say and when to say it”) to form a coherent high- level logical flow, and (2) appropriately reflect the BART Outputs (1) Monied interests will have a large influence in elections. (2) Corporations will be easily manipulated and controlled. (3) Public funding of elections would make our government far less corrupt. Statement: I think public funding of elections could solve many of our political problems. CMV. Guidance Keyphrases: influence; government; election; measure; monied interest; corporation; public funding, corruption Human Argument (1) Unfortunately, public funding for elections would be easy for corporations to tap into. (2) Also, monied interests have a large influence on our government. (3) Our government would have to be less corrupt than it is now for such measures to work successfully. 1 2 3 Content Planning Surface Realization ! Figure 1: Sample counter-arguments on Reddit Change- MyView. Given a statement and a set of unordered keyphrases as guidance talking points, BART generates an incoherent output. In contrast, human writer conducts content planning and keyphrase selection for each sentence to form a coherent counter-argument. text plans into final outputs …

Content truncated to 500 words