Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 259–269 July 13-14, 2023 ©2023 Association for Computational Linguistics Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap Qiwei Peng David Weir Julie Weeds University of Sussex Brighton, UK {qiwei.peng, d.j.weir, j.e.weeds}@sussex.ac.uk Abstract Paraphrase detection is useful in many natural language understanding applications. Current works typically formulate this problem as a sentence pair binary classification task. How- ever, this setup is not a good fit for many of the intended applications of paraphrase models. In particular, such applications often involve finding the closest paraphrases of the target sentence from a group of candidate sentences where they exhibit different degrees of seman- tic overlap with the target sentence. To apply models to this paraphrase retrieval scenario, the model must be sensitive to the degree to which two sentences are paraphrases of one another. However, many existing datasets ignore and fail to test models in this setup. In response, we propose adversarial paradigms to create evalu- ation datasets, which could examine the sensi- tivity to different degrees of semantic overlap. Empirical results show that, while paraphrase models and different sentence encoders appear successful on standard evaluations, measuring the degree of semantic overlap still remains a big challenge for them. 1 Introduction Detecting paraphrases is useful in many natural lan- guage understanding applications, such as question answering (Yin et al., 2015; Gan and Ng, 2019), fact checking (Jiang et al., 2020), and text summari- sation (Kry´sci´nski et al., 2018, 2019). Researchers have constructed paraphrase identification bench- marks, typically formulating the problem as a sen- tence pair classification task (Dolan and Brockett, 2005; Lan et al., 2017; Iyer et al., 2017; Zhang et al., 2019b). Sentence pairs that have the same or largely equivalent semantics are considered as paraphrases of each other (Androutsopoulos and Malakasiotis, 2010; Bhagat and Hovy, 2013). For example: a) More than half of the songs were purchased as albums, Apple said. b) Apple noted that half the songs were pur- chased as part of albums. Not only is it unclear what the criteria is for determining when a sentence pair has sufficiently similar semantics to be considered paraphrases, but as Chen et al. (2020) point out, the standard para- phrase classification task is not a good fit for many of the intended applications of paraphrase models. In particular, such applications are often retrieval tasks that involve finding the closest paraphrases of some target sentences within a set of documents, where candidate sentences exhibit different degrees of semantic overlap with the target sentence. To apply models to a paraphrase retrieval scenario, a paraphrase model must be sensitive to the degree to which two sentences are paraphrases of one an- other. We use the term partial paraphrase to refer to situations where a sentence pair has some overlap in meaning, but this can range from nearly exact paraphrases to pairs that share very little meaning. An example of an intermediate case is given …

Content truncated to 500 words