Sentence-Permuted Paragraph Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5051–5062 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5051 Sentence-Permuted Paragraph Generation Wenhao Yu†, Chenguang Zhu‡, Tong Zhao†, Zhichun Guo†, Meng Jiang† †University of Notre Dame ‡Microsoft Cognitive Services Research †{wyu1, tzhao2, zguo5, mjiang2}@nd.edu ‡ chezhu@microsoft.com Abstract Generating paragraphs of diverse contents is important in many applications. Existing gen- eration models produce similar contents from homogenized contexts due to the fixed left-to- right sentence order. Our idea is permuting the sentence orders to improve the content diver- sity of multi-sentence paragraph. We propose a novel framework PermGen whose objective is to maximize the expected log-likelihood of output paragraph distributions with respect to all possible sentence orders. PermGen uses hi- erarchical positional embedding and designs new procedures for both training phase and inference phase. Experiments on three para- graph generation benchmarks demonstrate Per- mGen generates more diverse outputs with a higher quality than existing models. 1 Introduction Paragraph generation is an important yet challeng- ing task. It requires a model to generate informa- tive and coherent long text that consists of mul- tiple sentences from free-format sources such as a topic statement or some keywords (Guo et al., 2018). Typical paragraph generation tasks include story generation (Fan et al., 2018), news genera- tion (Leppänen et al., 2017), scientific paper gener- ation (Koncel-Kedziorski et al., 2019), etc. Recent advances in natural language generation models such as Transformer (Vaswani et al., 2017) and BART (Lewis et al., 2020) have demonstrated at- tractive performance of generating text paragraphs. An important desired property of model- generated paragraphs is diversity – given the same source, an intelligent model is expected to create a variety of paragraphs in terms of content, seman- tic style, and word variability (Li et al., 2016; Ip- polito et al., 2019). For example, a story generation § Our code and output files are available at https:// github.com/wyu97/permgen. 0.00 0.05 0.10 0.15 0.20 0.25 S1 S2 S3 S4 S5 Diversity (→) Sentence index BART PermGen p=0.01 0.229 0.237 0.20 0.21 0.22 0.23 0.24 0.25 Accuracy (→) Paragraph Figure 1: Left: Diversity of each generated story sen- tence at different positions (1 to 5) in ROCStories’ test set, measured by averaged 1-Self-BLEU (Zhu et al., 2018). Our PermGen produces contents of higher di- versity at all positions, while BART (dashed line) pro- duces diverse outputs only at the end of story. With p- value<0.01, PermGen has higher diversity than the the grey line. Right: PermGen outperforms BART in the accuracy of generated stories measured by BLEU-4. model should narrate a plot with different story- lines (Clark et al., 2018); a scientific paper genera- tion model should suggest diverse contents to spark new ideas (Wang et al., 2019). In order to create diversity, controllable methods (Zhao et al., 2017; Cho et al., 2019; Yu et al., 2020) used additional inputs (e.g., aspects, styles). Sampling decoding algorithms (Radford et al., 2019; Holtzman et al., 2020) searched next tokens widely from a vocab- …

Content truncated to 500 words