POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8649–8670, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8649 POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training Yizhe Zhang1∗ Guoyin Wang2∗ † Chunyuan Li1 Zhe Gan1 Chris Brockett1 Bill Dolan1 1Microsoft Research, Redmond, WA, USA 2Amazon Alexa AI, Seattle, WA, USA {yizzhang,chunyl,zhe.gan,chrisbkt,billdol}@microsoft.com, guoyiwan@amazon.com Abstract Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learn- ing and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical con- straints. To address this challenge, we present POINTER1, a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progres- sively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is com- pleted. The resulting coarse-to-fine hierar- chy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based ob- jective on a 12GB Wikipedia dataset, and fine- tune it on downstream hard-constrained gen- eration tasks. Non-autoregressive decoding yields an empirically logarithmic time com- plexity during inference time. Experimental results on both News and Yelp datasets demon- strate that POINTER achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research 2. 1 Introduction Real-world editorial assistant applications must often generate text under specified lexical con- straints, for example, convert a meeting note with key phrases into a concrete meeting summary, re- cast a user-input search query as a fluent sentence, generate a conversational response using ground- ing facts (Mou et al., 2016), or create a story using a pre-specified set of keywords (Fan et al., 2018; Yao et al., 2019; Donahue et al., 2020). ∗These authors contributed equally to this work. †Work was done while Guoyin was at Microsoft. 1PrOgressive INsertion-based TransformER 2https://github.com/dreasysnail/POINTER Generating text under specific lexical constraints is challenging. Constrained text generation broadly falls into two categories, depending on whether inclusion of specified keywords in the output is mandatory. In soft-constrained generation (Qin et al., 2019; Tang et al., 2019), keyword-text pairs are typically first constructed (sometimes along with other conditioning information), and a con- ditional text generation model is trained to cap- ture their co-occurrence, so that the model learns to incorporate the constrained keywords into the generated text. While soft-constrained models are easy to design, even remedied by soft enforcing algorithms such as attention and copy mechanisms (Bahdanau et al., 2015; Gu et al., 2016; Chen et al., 2019a), keywords are still apt to be lost during gen- eration, especially when multiple weakly correlated keywords must be included. Hard-constrained generation (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et al., 2019; Miao et al., 2019; Welleck et al., 2019), on the other hand, requires that all the lexical constraints be present in the output sentence. This approach typi- cally involves sophisticated design of network ar- chitectures. Hokamp and Liu (2017) construct a …

Content truncated to 500 words