Effective Unsupervised Constrained Text Generation based on Perturbed Masking

Findings of the Association for Computational Linguistics: ACL 2022, pages 1417 - 1427 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Effective Unsupervised Constrained Text Generation based on Perturbed Masking Yingwen Fu1,2∗, Wenjie Ou2,∗†, Zhou Yu3, Yue Lin2 1Guangdong University of Foreign Studies, Guangzhou, China 2NetEase Games AI Lab, Guangzhou, China 3Columbia University 120201010002@gdufs.edu.cn, 3zy2461@columbia.edu, 2{ouwenjie, gzlinyue}@corp.netease.com Abstract Unsupervised constrained text generation aims to generate text under a given set of constraints without any supervised data. Current state-of- the-art methods stochastically sample edit po- sitions and actions, which may cause unneces- sary search steps. In this paper, we propose PMCTG to improve effectiveness by searching for the best edit position and action in each step. Specifically, PMCTG extends perturbed mask- ing technique to effectively search for the most incongruent token to edit. Then it introduces four multi-aspect scoring functions to select edit action to further reduce search difficulty. Since PMCTG does not require supervised data, it could be applied to different generation tasks. We show that under the unsupervised setting, PMCTG achieves new state-of-the-art results in two representative tasks, namely keywords- to-sentence generation and paraphrasing. 1 Introduction Constrained text generation is the task of generat- ing text that satisfies a given set of constraints, and it serves many real-world text generation applica- tions, such as dialogue generation (Li et al., 2016) and summarization (See et al., 2017). There are broadly two types of constraints: Hard constraints such as including a set of given words or phrases in the generated text. Example 1 in Table 1 shows that the keywords “You” and “beautiful” must oc- cur in the generated sentence. Soft constraints such as acquiring the generated text to be semantically similar to the original text. Example 2 in Table 1 shows a pair of paraphrases where “What are the effective ways to learn cs?” and “How to learn cs effectively?” share a similar meaning. Conventional approaches model the task in an encoding-decoding paradigm with a supervised setting (Prakash et al., 2016; Gupta et al., 2018). ∗‘ Equal contribution. This work was conducted when Yingwen Fu was interning at NetEase Games AI Lab. †‘ Corresponding author. No. Original Text Generated Text 1 You, beautiful You are so beautiful . 2 How to learn cs effectively? What are the effective ways to learn cs? Table 1: Examples on constrained text generation. However, these methods have certain shortcom- ings for two constrained generation tasks. For hard constrained text generation, without external con- strained means, these methods are difficult to guar- antee that the generated text can satisfy all con- straints. For soft constrained one, conventional methods treat it as a machine translation (MT) task (Sutskever et al., 2014) and require massive par- allel supervised data for training. Unfortunately, constructing such datasets is resource-intensive. In addition, domain-specific supervised models may be difficult to transfer to new domains. (Li et al., 2019). Recently, unsupervised text generation is pro- posed to address the above challenges. There are mainly two research directions: Beam search-based method aims to …

Content truncated to 500 words