Parallel Refinements for Lexically Constrained Text Generation with BART

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8653–8666 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8653 Parallel Refinements for Lexically Constrained Text Generation with BART Xingwei He Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China hexingwei15@gmail.com Abstract Lexically constrained text generation aims to control the generated text by incorporating some pre-specified keywords into the output. Previous work injects lexical constraints into the output by controlling the decoding pro- cess or refining the candidate output iteratively, which tends to generate generic or ungram- matical sentences, and has high computational complexity. To address these challenges, we propose Constrained BART (CBART) for lex- ically constrained text generation. CBART leverages the pre-trained model BART and transfers part of the generation burden from the decoder to the encoder by decomposing this task into two sub-tasks, thereby improving the sentence quality. Concretely, we extend BART by adding a token-level classifier over the encoder, aiming at instructing the decoder where to replace and insert. Guided by the en- coder, the decoder refines multiple tokens of the input in one step by inserting tokens be- fore specific positions and re-predicting tokens with low confidence. To further reduce the in- ference latency, the decoder predicts all tokens in parallel. Experiment results on One-Billion- Word and Yelp show that CBART can generate plausible text with high quality and diversity while significantly accelerating inference. 1 Introduction Controllable text generation aims to generate text in a controlled way, such as transferring text style (Shen et al., 2017; Fu et al., 2018; Li et al., 2018; Xu et al., 2018) and generating text with control codes (Keskar et al., 2020). Lexically constrained text generation requires that the given keywords must appear in the output, which can be applied to incorporating keywords into a dialog response (Mou et al., 2016), creating a story with keywords (Fan et al., 2018), generating advertisements for products (Miao et al., 2019) and writing a concrete meeting summary based on several key phrases. Cons lovely, time, forever, try Step 1 this lovely experience time and forever to try . Step 2 this was lovely experience ! time and it forever to try this . Step 3 this was a lovely experience ! first time here and it took forever to try this place . Table 1: An example from the Yelp test set demonstrates how CBART generates text with lexical constraints. In each refinement step, tokens underlined are newly generated. To generate sentences with keywords, Mou et al. (2015) proposed a backward and forward language model (B/F-LM). Liu et al. (2019a) applied adver- sarial learning (Goodfellow et al., 2014) to B/F-LM. Both models are limited to generating text with one keyword. To incorporate multiple keywords into machine translation, Hokamp and Liu (2017) pro- posed grid beam search (GBS) by adding an addi- tional constrained dimension to beam search. How- ever, GBS does not consider the future lexical con- straints when generating previous tokens, thereby degrading the …

Content truncated to 500 words