Towards Content Transfer through Grounded Text Generation

Proceedings of NAACL-HLT 2019, pages 2622–2632 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2622 Towards Content Transfer through Grounded Text Generation Shrimai Prabhumoye Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15219 sprabhum@andrew.cmu.edu Chris Quirk, Michel Galley Microsoft Research One Microsoft Way Redmond, WA 98052 {chrisq,mgalley}@microsoft.com Abstract Recent work in neural generation has attracted significant interest in controlling the form of text, such as style, persona, and politeness. However, there has been less work on control- ling neural text generation for content. This paper introduces the notion of Content Trans- fer for long-form text generation, where the task is to generate a next sentence in a docu- ment that both fits its context and is grounded in a content-rich external textual source such as a news story. Our experiments on Wikipedia data show significant improvements against competitive baselines. As another contribution of this paper, we release a benchmark dataset of 640k Wikipedia referenced sentences paired with the source articles to encourage explo- ration of this new task. 1 Introduction Recent work in neural natural language genera- tion (NLG) has witnessed a growing interest in controlling text for various form-related and lin- guistic properties, such as style (Ficler and Gold- berg, 2017), affect (Ghosh et al., 2017), politeness (Sennrich et al., 2016), persona (Li et al., 2016b) voice (Yamagishi et al., 2016), grammatical cor- rectness (Ji et al., 2017), and length (Kikuchi et al., 2016). This trend offers the promise of empow- ering existing authoring tools such as Grammarly, Google Smart Compose, and Microsoft Word with the ability to control a much greater variety of tex- tual properties, which are currently mostly lim- ited to grammar, spelling, word choice, and wordi- ness. What has been relatively less explored in neural NLG research is the ability to control the generation of a current sentence not only in its form, but also its content.1 Consider for example Fig. 1, which illustrates a situation where an au- thor edits a document (here a Wikipedia article), 1Historically, NLG has focused on generation from struc- tured content such as a database or semantic representation, but this paper is interested in generation from free-form text. Figure 1: Example of content transfer: Given existing curated text (yellow) and a document with additional relevant information (green), the task is to update the curated text (orange) to reflect the most salient updates. and the goal is to generate or suggest a next sen- tence (shown in orange) to the author. This type of unconstrained, long-form text generation task (Mostafazadeh et al., 2016; Fan et al., 2018) is of course extremely difficult. Free-form generation can easily go astray due to two opposing factors. On one hand, ensuring that the generated output is of relatively good quality often comes at the cost of making it bland and devoid of factual con- tent (Li et al., 2016a). On the other hand, exist- ing techniques can help steer neural models away 2623 from blandness in order to produce …

Content truncated to 500 words