Towards Better Characterization of Paraphrases

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8592 - 8601 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Towards Better Characterization of Paraphrases Timothy Liu1 De Wen Soh1 1Singapore University of Technology and Design timothy_liu@mymail.sutd.edu.sg, dewen_soh@sutd.edu.sg Abstract To effectively characterize the nature of para- phrase pairs without expert human annotation, we propose two new metrics: word position deviation (WPD) and lexical deviation (LD). WPD measures the degree of structural alter- ation, while LD measures the difference in vocabulary used. We apply these metrics to better understand the commonly-used MRPC dataset and study how it differs from PAWS, another paraphrase identification dataset. We also perform a detailed study on MRPC and propose improvements to the dataset, show- ing that it improves generalizability of models trained on the dataset. Lastly, we apply our metrics to filter the output of a paraphrase gen- eration model and show how it can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models. 1 Introduction A robust understanding of semantic meaning, de- spite variances in sentence expression, is an integral part of natural language processing (NLP) tasks. However, many existing NLP models exhibit short- comings in understanding real-world variations in natural language. These models are often over- reliant on learned spurious correlations resulting in poor generalization (Sanchez et al., 2018; Mc- Coy et al., 2019). This problem is challenging to address since it is difficult to distinguish spurious correlations from useful features (Gardner et al., 2021). One way of improving the performance and ro- bustness of NLP model is to increase the size of the dataset (Hestness et al., 2017). It is possible to do so in an efficient manner through data augmen- tation, or the process of generating new data out of existing examples, thus creating more training data or test cases (Feng et al., 2021; Chen et al., 2021). This would also enhance the capability to detecting error in a wide range of NLP systems. We can also condition language models to generate paraphrases of input sentences (Witteveen and Andrews, 2019) through the use of large language models such as GPT (Radford et al., 2019). However, commonly used paraphrase datasets and paraphrase generation techniques that rely on such datasets suffer from several shortfalls, such as being noisy due to loose labelling in these datasets and lack of accurate, con- trollable generation. In this paper, we make three key contributions to address this issue. Firstly, we propose two new metrics for better understanding of paraphrase pairs: word position deviation and lexical deviation. We show, with ex- amples, how these metrics are more effective at quantitatively capturing the linguistic characteris- tics of paraphrase pair than existing methods such as ROGUE-L, SELF-BLEU and edit distance. Secondly, we apply the proposed metrics to bet- ter understand the commonly used Microsoft Re- search Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005) dataset. We also study how MRPC differs from Paraphrase Adversaries from Word …

Content truncated to 500 words