Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6533–6543, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6533 Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task Dongyeop Kang∗ dongyeopk@berkeley.edu UC Berkeley Eduard Hovy hovy@cs.cmu.edu Carnegie Mellon University Abstract Despite the recent success of contextualized language models on various NLP tasks, lan- guage model itself cannot capture textual co- herence of a long, multi-sentence document (e.g., a paragraph). Humans often make structural decisions on what and how to say about before making utterances. Guiding sur- face realization with such high-level decisions and structuring text in a coherent way is es- sentially called a planning process. Where can the model learn such high-level coher- ence? A paragraph itself contains various forms of inductive coherence signals called self-supervision in this work, such as sentence orders, topical keywords, rhetorical structures, and so on. Motivated by that, this work pro- poses a new paragraph completion task PAR- COM; predicting masked sentences in a para- graph. However, the task suffers from predict- ing and selecting appropriate topical content with respect to the given context. To address that, we propose a self-supervised text plan- ner SSPlanner that predicts what to say first (content prediction), then guides the pretrained language model (surface realization) using the predicted content. SSPlanner outperforms the baseline generation models on the paragraph completion task in both automatic and human evaluation. We also find that a combination of noun and verb types of keywords is the most effective for content selection. As more num- ber of content keywords are provided, overall generation quality also increases. 1 Introduction One may think textual coherence can be achieved from a gigantic language model trained on massive data. This might be true in simple cases, such as generating short replies (Kannan et al., 2016), but not in a long, multi-sentence generation. This is ∗This work was done while DK was at CMU. mainly because per-word predictions from the au- toregressive models can not capture the long-term flow of text, while humans often make structural decisions on what and how to say about before they speak (Byrne, 1979; McKeown, 1985; Hovy, 1990; Swan, 2002; Kang, 2020). Guiding the surface- level realization with such high-level decisions and coherently structuring output text is called a plan- ning process. Where can the model learn such high-level de- cisions related to long-term coherence? A written paragraph itself can be a pot of golden resources, containing various forms of inductive coherence signals. Different types of coherence signals in a paragraph have been studied and used in many dif- ferent ways: a sequence of words or sentences (De- vlin et al., 2019; Radford et al., 2019), a discourse structure of a text (Appelt, 1982; Hovy, 1991; Kang et al., 2019), an order of sentences (Chambers and Jurafsky, 2008; Barzilay and Lapata, 2008), topic introduction, co-reference, a sequence of events (Tomkins, 1978; Schank and Abelson, 2013), and more. In this work, we primarily focus on the effect of topical content in …

Content truncated to 500 words