Enabling Language Models to Fill in the Blanks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492–2501 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2492 Enabling Language Models to Fill in the Blanks Chris Donahue Stanford University Mina Lee Stanford University {cdonahue,minalee,pliang}@cs.stanford.edu Percy Liang Stanford University Abstract We present a simple approach for text infill- ing, the task of predicting missing spans of text at any position in a document. While infill- ing could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling—a special case of infilling where text is predicted at the end of a document. In this paper, we aim to ex- tend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can en- able LMs to infill entire sentences effectively on three different domains: short stories, sci- entific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine- generated in the domain of short stories. 1 Introduction Text infilling is the task of predicting missing spans of text which are consistent with the preceding and subsequent text.1 Systems capable of infilling have the potential to enable rich applications such as assisting humans in editing or revising text (Shih et al., 2019), connecting fragmented ideas (AI21, 2019), and restoring ancient documents (Assael et al., 2019). Rather than targeting a particular application, our goal here is to provide a general, flexible, and simple infilling framework which can convincingly infill in a variety of domains. A special case of infilling is language modeling: predicting text given preceding but not subsequent text.2 Language models are (1) capable of generat- 1Text infilling is a generalization of the cloze task (Taylor, 1953)—cloze historically refers to infilling individual words. 2In this paper, language modeling always refers to ordinary LMs, i.e., “unidirectional,” “autoregressive,” or “left-to-right.” She ate leftover pasta for lunch. She ate [blank] for [blank]. leftover pasta [answer] lunch [answer] Data Input Target Our Infilling Framework She ate [blank] for [blank]. She ate leftover pasta for lunch. Infilling Task Input Output Train Language Model Infilling Input [sep] Target Data Input [sep] Target Output Figure 1: We consider the task of infilling, which takes incomplete text as input and outputs completed text. To tackle this task, our framework constructs training ex- amples by masking random spans to generate pairs of inputs (text with blanks) and targets (answers for each blank). We then train unidirectional language mod- els on the concatenation of each pair. Once trained, a model takes text input with blanks, predicts the an- swers, and then combines them to produce the output. ing remarkably coherent text (Zellers et al., 2019; See et al., 2019), (2) efficient at generating text, and (3) conceptually simple, but cannot infill ef- fectively as …

Content truncated to 500 words