ParaTag: A Dataset of Paraphrase Tagging for Fine-Grained Labels, NLG Evaluation, and Data Augmentation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7111 - 7122 December 7-11, 2022 ©2022 Association for Computational Linguistics ParaTag : A Dataset of Paraphrase Tagging for Fine-Grained Labels, NLG Evaluation, and Data Augmentation Shuohang Wang*, Ruochen Xu*, Yang Liu*, Chenguang Zhu, Michael Zeng Microsoft Azure Cognitive Services Research {shuowa, ruox, yaliu10, chezhu, nzeng}@microsoft.com Abstract Paraphrase identification has been formulated as a binary classification task to decide whether two sentences hold a paraphrase relationship. Existing paraphrase datasets only annotate a binary label for each sentence pair. However, after a systematical analysis of existing para- phrase datasets, we found that the degree of paraphrase cannot be well characterized by a single binary label. And the criteria of para- phrase are not even consistent within the same dataset. We hypothesize that such issues would limit the effectiveness of paraphrase models trained on these data. To this end, we pro- pose a novel fine-grained paraphrase annota- tion schema that labels the minimum spans of tokens in a sentence that don’t have the cor- responding paraphrases in the other sentence. Under this setting, we frame paraphrasing as a sequence tagging task. We collect 30k sen- tence pairs in English with the new annota- tion schema, resulting in the ParaTag dataset. In addition to reporting baseline results on ParaTag using state-of-art language models, we show that ParaTag is especially useful for train- ing an automatic scorer for language genera- tion evaluation. Finally, we train a paraphrase generation model from ParaTag and achieve better data augmentation performance on the GLUE benchmark than other public paraphras- ing datasets.1 1 Introduction Paraphrase identification and paraphrase genera- tion are important problems in natural language processing. They are the essential tasks to ver- ify whether machine learning models genuinely understand the text. A lot of research (Das and Smith, 2009; Yin et al., 2016; Devlin et al., 2019; He et al., 2021) is devoted to identifying paraphrase between sentences, on various annotated datasets such as Microsoft Research Paraphrase Corpus * Equal contribution. 1Data and code: https://github.com/microsoft/ParaTag M-1 However, commercial use of the 2.6.0 kernel is still months off for most customers. M-2 Commercial releases of the 2.6 kernel by major Linux distributors still remain months away. G-1 It covers some of the same ground as Spotlight but has nothing new to add. G-2 It revisits the same topic as Spotlight, but does not say anything new. G-3 Heavy rainstorms have caused floods in the area every year since 2003. G-4 Since 2003, the area has been hit every year by heavy rainstorms and floods. Table 1: Examples of tags in ParaTag . M: sentences from MRPC. G: sentences generated by GPT-3. The phrases in bold are judged by human annotators as not being paraphrased by the other sentence. Note that the first sentence pair is originally labeled as paraphrase in MRPC. However, the fine-grained annotation indicates that not all the information is paraphrased. (MRPC) (Dolan and Brockett, 2005), Quora Ques- tion Pairs Dataset (QQP)2(Wang et …

Content truncated to 500 words