Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries

Proceedings of the 29th International Conference on Computational Linguistics, pages 6392–6402 October 12–17, 2022. 6392 Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries Xiaofei Sun1,2, Zijun Sun2, Yuxian Meng2, Jiwei Li1,2 and Chun Fan3 1Zhejiang University, 2Shannon.AI, 3Computer Center, Peking University 3National Biomedical Imaging Center, Peking University, 3Peng Cheng Laboratory xiaofei_sun@zju.edu.cn, jiwei_li@shannonai.com Abstract The difficulty of generating coherent long texts lies in the fact that existing models overwhelm- ingly focus on predicting local words, and can- not make high level plans on what to generate or capture the high-level discourse dependen- cies between chunks of texts. Inspired by hu- man writing processes, where a list of bullet points or a catalog is first outlined, and then each bullet point is expanded to form the whole article, we propose SOE, a pipelined system that involves of summarizing, outlining and elaborating for long text generation: the model first outlines the summaries for different seg- ments of long texts, and then elaborates on each bullet point to generate the correspond- ing segment. To avoid the labor-intensive pro- cess of summary soliciting, we propose the re- construction strategy, which extracts segment summaries in an unsupervised manner by se- lecting its most informative part to reconstruct the segment. The proposed generation system comes with the following merits: (1) the summary provides high-level guidance for text generation and avoids the local minimum of individual word predictions; (2) the high-level discourse depen- dencies are captured in the conditional depen- dencies between summaries and are preserved during the summary expansion process and (3) additionally, we are able to consider signifi- cantly more contexts by representing contexts as concise summaries. Extensive experiments demonstrate that SOE produces long texts with significantly better quality, along with faster convergence speed. 1 Introduction Despite that recent large-scale pretrained language models (PLMs) (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019) are able to produce high- quality passages that can be hardly recognized by humans (Zellers et al., 2019), most of the generated “good” texts are within very limited length, e.g. hun- dreds of tokens for most cases (Yan et al., 2020), thus generating coherent long texts remains a chal- lenge (Radford et al., 2019; Tan et al., 2020). The difficulty lies in the fact that existing models gen- erate texts in a word-by-word manner: predicting each subsequent token given its preceding contexts. This word-by-word strategy overwhelmingly fo- cuses on predicting local words, and cannot make high level plans on what to generate. The strat- egy results in the fact that long texts generated by current models are usually repetitive, generic and self-contradictory (Shen et al., 2019). To address these issues, the coarse-to-fine gener- ation strategy is proposed (Fan et al., 2018; Xu et al., 2018; Yao et al., 2019; Mao et al., 2019). In coarse-to-fine generation, a list of keywords or a short prompt is first generated, serving as a sum- mary of the original text. The prompt is then fed to a seq2seq model as …

Content truncated to 500 words