Predicate-Argument Based Bi-Encoder for Paraphrase Identification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5579 - 5589 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Predicate-Argument Based Bi-Encoder for Paraphrase Identification Qiwei Peng University of Sussex Brighton, UK qiwei.peng@sussex.ac.uk David Weir University of Sussex Brighton, UK d.j.weir@sussex.ac.uk Julie Weeds University of Sussex Brighton, UK j.e.weeds@sussex.ac.uk Yekun Chai Baidu, Inc. Beijing, China chaiyekun@baidu.com Abstract Paraphrase identification involves identifying whether a pair of sentences express the same or similar meanings. While cross-encoders have achieved high performances across sev- eral benchmarks, bi-encoders such as SBERT have been widely applied to sentence pair tasks. They exhibit substantially lower com- putation complexity and are better suited to symmetric tasks. In this work, we adopt a bi- encoder approach to the paraphrase identifica- tion task, and investigate the impact of explic- itly incorporating predicate-argument informa- tion into SBERT through weighted aggrega- tion. Experiments on six paraphrase identifica- tion datasets demonstrate that, with a minimal increase in parameters, the proposed model is able to outperform SBERT/SRoBERTa signifi- cantly. Further, ablation studies reveal that the predicate-argument based component plays a significant role in the performance gain. 1 Introduction Paraphrases are sentences that express the same or similar meanings with different wording (Bhagat and Hovy, 2013). Paraphrase pairs are either fully or largely semantically equivalent. For example: a) Marriage equality law passed in Rhode Island b) Rhode Island becomes the 10th state to enact marriage equality It is generally considered to be a symmetric task where the paraphrase relation holds in both direc- tions (Bhagat and Hovy, 2013; Yang et al., 2019). Since word order and sentence structure are cru- cial in determining sentence meaning, effective paraphrase models must be structure-aware and word order sensitive. In light of this, paraphrase datasets have been created that are specifically de- signed to encourage models to consider structural differences (Xu et al., 2015; Zhang et al., 2019b). For example, PIT2015 (Xu et al., 2015) consists of paraphrase pairs that are lexically diverse and non-paraphrase pairs that are lexically similar but semantically dissimilar. There are generally two pre-trained based ap- proaches for sentence pair tasks such as paraphrase identification. The first is the cross-encoder ap- proach, which involves concatenating the two in- put sentences and performing full-attention over the input. The second is the bi-encoder approach, which adopts a conjoined twin network structure and maps each sentence onto separate represen- tations, which can then be compared using sim- ilarity measures such as cosine. Though typical cross-encoders like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b) have set state- of-the-art performance on various sentence pair tasks (Zhang et al., 2021; Xia et al., 2021), they still face challenges from both extreme computa- tional overhead for many use cases (Reimers and Gurevych, 2019; Thakur et al., 2021) and incon- sistent predictions (ranging from 2.66% to 8.46% depending on specific datasets) when dealing with symmetric tasks (Chen et al., 2020). In contrast, a bi-encoder approach such as Sentence-BERT (SBERT) (Reimers …

Content truncated to 500 words