Unsupervised Paraphrasing with Pretrained Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5136–5150 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5136 Unsupervised Paraphrasing with Pretrained Language Models Tong Niu Semih Yavuz Yingbo Zhou Nitish Shirish Keskar Huan Wang Salesforce Research {tniu, syavuz, yingbo.zhou, nkeskar, huan.wang, cxiong}@salesforce.com Caiming Xiong Abstract Paraphrase generation has benefited exten- sively from recent progress in the designing of training objectives and model architectures. However, previous explorations have largely focused on supervised methods, which require a large amount of labeled data that is costly to collect. To address this drawback, we adopt a transfer learning approach and propose a training pipeline that enables pre-trained lan- guage models to generate high-quality para- phrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dy- namic Blocking (DB). To enforce a surface form dissimilar from the input, whenever the language model emits a token contained in the source sequence, DB prevents the model from outputting the subsequent source token for the next generation step. We show with automatic and human evaluations that our ap- proach achieves state-of-the-art performance on both the Quora Question Pair (QQP) and the ParaNMT datasets and is robust to domain shift between the two datasets of distinct distri- butions. We also demonstrate that our model transfers to paraphrasing in other languages without any additional finetuning. 1 Introduction Paraphrase generation restates text input in a differ- ent surface form while preserving its semantics. It has various applications on downstream NLP tasks including text summarization (Cao et al., 2016), se- mantic parsing (Berant and Liang, 2014), as well as diversifying text generation for user-facing systems such as chatbots. To evaluate model robustness, a paraphraser can be used to generate adversarial examples, which also serve as augmented data to train the target neural networks (Iyyer et al., 2018a). Besides, paraphrasing queries makes Question An- swering systems more likely to match with key- words in a knowledge base (Fader et al., 2014; Yin et al., 2015). Language Model Task-adaptive Training Unsupervised Data Synthetic Parallel Data Corrupt and Shuffle Generate with Dynamic Blocking Self-supervised Model Self-supervision Data Task-adapted Model Self-supervised Training Figure 1: Training pipeline of our paraphrasing model. We first train a task-adapted model with a denoising objective so that it is able to reconstruct input text. We then use Dynamic Blocking (DB) to generate pseudo- pairs of paraphrasing data. Finally, the generated data is used to train the self-supervised model. However, it is expensive to annotate para- phrases, resulting in only a few human-labeled datasets. The existing ones are either small-scale like MRPC (Dolan and Brockett, 2005), or of closed domains like QQP1 which consists entirely of questions. Consequently, previous work ex- plored automatically (hence noisily) annotated datasets such as PIT-2015 (Xu et al., 2013), Twit- ter URL Paraphrase Corpus (Lan et al., 2017), ParaNMT (Wieting and Gimpel, 2018), and Para- Bank (Hu et al., 2019), or re-purposed datasets including MSCOCO (Lin et al., 2014) and WikiAn- swers (Fader et …

Content truncated to 500 words