ConRPG: Paraphrase Generation using Contexts as Regularizer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2551–2562 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2551 ConRPG: Paraphrase Generation using Contexts as Regularizer Yuxian Meng♣, Xiang Ao▲, Qing He▲, Xiaofei Sun♣ Qinghong Han♣, Fei Wu♦, Chun fan♠⋆▼ and Jiwei Li♦♣ ♦Zhejiang University ♠Computer Center of Peking University, ⋆Peng Cheng Laboratory ▼National Biomedical Imaging Center, Peking University ▲ Key Lab of Intelligent Information Processing of Chinese Academy of Sciences ♣Shannon.AI {yuxian_meng, xiaofei_sun, qinghong_han, jiwei_li}@shannonai.com {aoxiang, heqing}@ict.ac.cn, fanchun@pku.edu.cn, wufei@zju.edu.cn Abstract A long-standing issue with paraphrase gen- eration is how to obtain reliable supervision signals. In this paper, we propose an unsu- pervised paradigm for paraphrase generation based on the assumption that the probabili- ties of generating two sentences with the same meaning given the same context should be the same. Inspired by this fundamental idea, we propose a pipelined system which consists of paraphrase candidate generation based on con- textual language models, candidate filtering us- ing scoring functions, and paraphrase model training based on the selected candidates. The proposed paradigm offers merits over ex- isting paraphrase generation methods: (1) us- ing the context regularizer on meanings, the model is able to generate massive amounts of high-quality paraphrase pairs; and (2) us- ing human-interpretable scoring functions to select paraphrase pairs from candidates, the proposed framework provides a channel for de- velopers to intervene with the data generation process, leading to a more controllable model. Experimental results across different tasks and datasets demonstrate that the effectiveness of the proposed model in both supervised and un- supervised setups. 1 Introduction Paraphrase generation (Prakash et al., 2016a; Cao et al., 2016; Ma et al., 2018; Wang et al., 2018) is the task of generating an output sentence which is semantically identical to a given input sentence but with variations in lexicon or syntax. It is a long-standing problem in the field of natural lan- guage processing (NLP) (McKeown, 1979; Meteer and Shaked, 1988; Quirk et al., 2004; Bannard and Callison-Burch, 2005a; Chen and Dolan, 2011) and has fundamental applications on end tasks such as semantic parsing (Berant and Liang, 2014), lan- guage model pretraining (Lewis et al., 2020) and question answering (Dong et al., 2017). A long-standing challenge with paraphrase genera- tion is to obtain reliable supervision signals. One way to resolve this issue is to manually annotate paraphrase pairs, which is both labor-intensive and expensive. Existing labeled paraphrase datasets (Lin et al., 2014; Fader et al., 2013; Lan et al., 2017) are either of small sizes or restricted in nar- row domains. For example, the Quora dataset1 contains 140K paraphrase pairs, the size of which is insufficient to build a large neural model. As an- other example, paraphrases in the larger MSCOCO (Lin et al., 2014) dataset are originally collected as image captions for object recognition, and re- purposed for paraphrase generation. The domain for the MSCOCO dataset is thus restricted to cap- tions depicting visual scenes. Unsupervised methods, such as reinforcement learning (Li et al., 2018; Siddique …

Content truncated to 500 words