Substance over Style: Document-Level Targeted Content Transfer

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6485–6504, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6485 Substance over Style: Document-Level Targeted Content Transfer Allison Hegel1∗ Sudha Rao2 Asli Celikyilmaz2 Bill Dolan2 1Lexion, Seattle, WA, USA 2Microsoft Research, Redmond, WA, USA allison@lexion.ai {sudhra,aslicel,billdol}@microsoft.com Abstract Existing language models excel at writing from scratch, but many real-world scenarios re- quire rewriting an existing document to fit a set of constraints. Although sentence-level rewrit- ing has been fairly well-studied, little work has addressed the challenge of rewriting an entire document coherently. In this work, we intro- duce the task of document-level targeted con- tent transfer and address it in the recipe do- main, with a recipe as the document and a di- etary restriction (such as vegan or dairy-free) as the targeted constraint. We propose a novel model for this task based on the generative pre- trained language model (GPT-2) and train on a large number of roughly-aligned recipe pairs.1 Both automatic and human evaluations show that our model out-performs existing methods by generating coherent and diverse rewrites that obey the constraint while remaining close to the original document. Finally, we ana- lyze our model’s rewrites to assess progress to- ward the goal of making language generation more attuned to constraints that are substantive rather than stylistic. 1 Introduction We often think that writing starts from a blank page, but in practice, writing often involves adapting an existing document to fit a new context. This might involve rewriting documentation written for a Mac so that it will apply to a PC, rewriting a lesson plan for a different grade level, or rewriting a product description to appeal to customers in multiple re- gions. Automating such rewriting is valuable but challenging, since it requires learning to make coor- dinated changes spanning an entire document while adhering to constraints that apply not to the style but to the substance of the document. ∗*Work done when the author was at Microsoft Research. 1https://github.com/microsoft/ document-level-targeted-content-transfer Figure 1: Document-level targeted content transfer in the recipe domain: given a hot cocoa recipe and the user constraint vegan, the task is to rewrite the recipe into a vegan hot cocoa recipe. We introduce the novel task of document-level targeted content transfer, defined as rewriting a document to obey a user-provided constraint result- ing in some systematic alteration of the document’s content. Success at this task involves both transfer and controlled generation at the document level. Prior work on controlled generation guides the out- put of a model using attribute classifiers (Dathathri et al., 2020) or control codes (Keskar et al., 2019), but we find that these models do not perform well on our transfer task (§4.1.2). In contrast, models built for the transfer task are generally trained at the sentence level (Hu et al., 2017b,a; Li et al., 2018; Rao and Tetreault, 2018; Syed et al., 2019). Document-level transfer has typically found suc- cess by rewriting each sentence independently (Maruf et al., 2019). However, …

Content truncated to 500 words