Blank Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5186–5198, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5186 Blank Language Models Tianxiao Shen∗ Victor Quach∗ Regina Barzilay Tommi Jaakkola MIT CSAIL {tianxiao, quach, regina, tommi}@csail.mit.edu Abstract We propose Blank Language Model (BLM), a model that generates sequences by dynami- cally creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks. The model can start from a single blank or partially completed text with blanks at specified locations. It iteratively de- termines which word to place in a blank and whether to insert new blanks, and stops gen- erating when no blanks are left to fill. BLM can be efficiently trained using a lower bound of the marginal data likelihood. On the task of filling missing text snippets, BLM signifi- cantly outperforms all other baselines in terms of both accuracy and fluency. Experiments on style transfer and damaged ancient text restora- tion demonstrate the potential of this frame- work for a wide range of applications.1 1 Introduction Neural language models have shown impressive performance across many applications such as ma- chine translation and summarization where the text is generated from scratch (Bahdanau et al., 2014; Rush et al., 2015). However, a broader set of text generation tasks — including text editing, informa- tion fusion, and ancient text restoration — requires the model to start with partially specified text and generate the missing fragments. In the general setup, the input document may have any number of missing spans, and each span may have an un- known number of missing tokens. To perform this text infilling task (Zhu et al., 2019), a model should: (1) provide fine-grained control over the generation location, (2) accommodate a variable number of missing tokens, and (3) respect both the preceding and following context. ∗Equal contribution 1Our code is available at https://github.com/ Varal7/blank_language_model They also have which . They also have ice cream which is really good . Figure 1: BLM fills in blanks of arbitrary length. Existing approaches focus on adapting left-to- right language models for text infilling. Intricate inference algorithms leveraging dynamic program- ming or gradient search are proposed to find the filling content that has a high likelihood within the surrounding context (Sun et al., 2017; Liu et al., 2019a; Zaidi et al., 2020). These methods make simplified Markov assumptions, require high de- coding time complexity, and cannot adapt to vari- able infilling length. Alternatively, Donahue et al. (2020) predict the concatenation of the infilling content, but do not guarantee that the output will match the number of missing spans in the input. In this work, we introduce the Blank Language Model (BLM), which uses a special “ ” symbol to control where tokens can be placed. The gener- ation of BLM follows the grammar of replacing a blank with a word and possibly adjoining blanks. By jointly modeling context and missing content, …

Content truncated to 500 words