Automatic Document Sketching: Generating Drafts from Analogous Texts

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2102–2113 August 1–6, 2021. ©2021 Association for Computational Linguistics 2102 Automatic Document Sketching: Generating Drafts from Analogous Texts Zeqiu Wu♦ Michel Galley♠ Chris Brockett♠ Yizhe Zhang♠ Bill Dolan♠ ♦University of Washington ♠Microsoft Research zeqiuwu1@washington.edu {mgalley,chrisbkt,yizzhang,billdol}@microsoft.com Abstract The advent of large pre-trained language mod- els has made it possible to make high-quality predictions on how to add or change a sentence in a document. However, the high branching factor inherent to text generation impedes the ability of even the strongest language models to offer useful editing suggestions at a more global or document level. We introduce a new task, DOCUMENT SKETCHING, which in- volves generating entire draft documents for the writer to review and revise. These drafts are built from sets of documents that overlap in form – sharing large segments of potentially reusable text – while diverging in content. To support this task, we introduce a Wikipedia- based dataset of analogous documents and in- vestigate the application of weakly supervised methods, including use of a transformer-based mixture of experts, together with reinforce- ment learning. We report experiments using automated and human evaluation methods and discuss relative merits of these models. 1 Introduction Large pre-trained language models such as T5 and GPT-3 (Raffel et al., 2019; Brown et al., 2020) have enabled impressive progress on a variety of natu- ral language generation tasks by producing fluent, coherent long texts (Rashkin et al., 2020; Zellers et al., 2019). While automated document-level gen- eration seems tantalizingly within reach, a high branching factor presents significant challenges in tailoring generated documents to the specific re- quirements of users. Topic drift and “hallucina- tion” of information are endemic to these models (Wiseman et al., 2017). These risks have ensured that end-user applications involving text generation (e.g., Smart Compose, Smart Reply, Grammarly) still require a human to remain in control of con- tent and are restricted to individual sentences or even smaller segments of text (Chen et al., 2019; Figure 1: The right side shows a sketch for writing the report of a future democratic national convention, generated from a pile of previous reports. Kannan et al., 2016; Alikaniotis and Raheja, 2019; Prabhumoye et al., 2019; Faltings et al., 2021). Can large generative language models be used to assist user writing at the document level while the user still controls the factual content? A possi- ble answer lies in the observation that a substantial portion of day-to-day writing involves some form of reuse. Similar documents (e.g., monthly reports, sales letters, job descriptions) are effectively re- cycled by changing those segments that need to be modified (Fig. 1).1 Moreover, documents con- taining analogous texts are often found collocated in repositories, a common practice for organiza- tions that manage professional documents.2 The high branching factor that impedes the applica- tion of long-form generation might thus be mit- igated if models were to exploit conventionally- structured analogous “reusable” texts to produce a “sketch” that captures prototypical document-level text structure. In …

Content truncated to 500 words