Gradient-guided Unsupervised Lexically Constrained Text Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8692–8703, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8692 Gradient-guided Unsupervised Lexically Constrained Text Generation Lei Sha Apple Inc. lei sha@apple.com Abstract Lexically-constrained generation requires the target sentence to satisfy some lexical con- straints, such as containing some specific words or being the paraphrase to a given sentence, which is very important in many real-world natural language generation appli- cations. Previous works usually apply beam- search-based methods or stochastic search- ing methods to lexically-constrained genera- tion. However, when the search space is too large, beam-search-based methods always fail to find the constrained optimal solution. At the same time, stochastic search methods al- ways cost too many steps to find the cor- rect optimization direction. In this paper, we propose a novel method G2LC to solve the lexically-constrained generation as an un- supervised gradient-guided optimization prob- lem. We propose a differentiable objective function and use the gradient to help deter- mine which position in the sequence should be changed (deleted or inserted/replaced by an- other word). The word updating process of the inserted/replaced word also benefits from the guidance of gradient. Besides, our method is free of parallel data training, which is flex- ible to be used in the inference stage of any pre-trained generation model. We apply G2LC to two generation tasks: keyword-to-sentence generation and unsupervised paraphrase gener- ation. The experiment results show that our method achieves state-of-the-art compared to previous lexically-constrained methods. 1 Introduction In many natural language generation applications, there are usually some constraints required to be satisfied by the generated sequences. The con- straints can be classified into two types: 1. Hard constraints: some specific words or phrases must occur in the target sentence. For example, the facts in abstractive summariza- tion (See et al., 2017a). In detail, when do- ing summarization, it is always required to keep some key information like the facts. So, the facts are hard constraints for the summa- rization generation. Another example is the keywords (name or topic) in dialogue gener- ation (Li et al., 2016). These keywords are usually determined by the context of a dia- logue, and are required to occur in the next utterance. 2. Soft constraints: such as that the target sen- tence must have a similar meaning to a given sentence. For example, the synonymous con- straint in paraphrase generation (Prakash et al., 2016; Li et al., 2019). Previous works for lexically-constrained genera- tion can be divided into two branches: enhanced beam search (Hokamp and Liu, 2017; Post and Vi- lar, 2018) and stochastic search (Miao et al., 2019; Liu et al., 2019). Among the various enhanced beam search methods, grid beam search (Hokamp and Liu, 2017) is the most representative approach, which proposed to add candidate sequences that meet the lexical constraints to the beam in each step to constrain the search space. Dynamic beam allocation methods (Post and Vilar, 2018; Hu et al., 2019) are the extension of grid beam search, …

Content truncated to 500 words