On the Evaluation Metrics for Paraphrase Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3178 - 3190 December 7-11, 2022 ©2022 Association for Computational Linguistics On the Evaluation Metrics for Paraphrase Generation Lingfeng Shen♠∗ Lemao Liu♡ Haiyun Jiang♡† Shuming Shi♡ ♠Department of Computer Science, Johns Hopkins University ♡Natural Language Processing Center, Tencent AI Lab lshen30@jh.edu {redmondliu,haiyunjiang,shumingshi}@tencent.com Abstract In this paper we revisit automatic metrics for paraphrase evaluation and obtain two findings that disobey conventional wisdom: (1) Reference-free metrics achieve better per- formance than their reference-based counter- parts. (2) Most commonly used metrics do not align well with human annotation. Un- derlying reasons behind the above findings are explored through additional experiments and in-depth analyses. Based on the exper- iments and analyses, we propose ParaScore, a new evaluation metric for paraphrase gen- eration. It possesses the merits of reference- based and reference-free metrics and explic- itly models lexical divergence. Based on our analysis and improvements, our proposed reference-based outperforms than reference- free metrics. Experimental results demon- strate that ParaScore significantly outperforms existing metrics. Our codes and toolkit are released in https://github.com/ shadowkiller33/ParaScore. 1 Introduction Paraphrase generation is a fundamental problem in natural language processing (NLP), which has been widely applied in versatile tasks, such as ques- tion answering (Dong et al., 2017; Lan and Xu, 2018; Gan and Ng, 2019; Abujabal et al., 2019), machine translation (Madnani et al., 2012; Apidi- anaki et al., 2018; Kajiwara, 2019), and semantic parsing (Herzig and Berant, 2019; Wu et al., 2021; Cao et al., 2020). In particular, paraphrase genera- tion is also an important module in many writing assistant systems, such as QuillBot1, Wordtune2 and Effidit3 (Shi et al., 2022). Recent years have witnessed rapid development in paraphrase gen- eration algorithms (Sun et al., 2021; Huang and ∗Work was done during an internship at Tencent AI Lab. †Corresponding author. 1https://quillbot.com 2https://www.wordtune.com/ 3https://effidit.qq.com Chang, 2021; Kumar et al., 2020). However, little progress has been made in the automatic evaluation of this task. It is even unclear which metric is more reliable among many widely used metrics. Most evaluation metrics used in previous para- phrase generation research are not designed for the task itself, but adopted from other evaluation tasks, such as machine translation (MT) and sum- marization. However, paraphrase evaluation is in- herently different from the evaluation of most other tasks, because a good paraphrase typically obeys two criteria (Gleitman and Gleitman, 1970; Chen and Dolan, 2011; Bhagat and Hovy, 2013): seman- tic similarity (Sim) and lexical divergence (Div). Sim means that the paraphrase maintains similar semantics to the input sentence, whereas Div re- quires that the paraphrase possesses lexical or syn- tactic differences from the input. In contrast, tasks like machine translation have no requirement for Div. It is therefore uncertain whether the metrics borrowed from other tasks perform well in para- phrase evaluation. In this paper, we revisit automatic metrics for paraphrase evaluation. We collect a list of popu- lar metrics used in recent researches (Kumar et al., 2019; Feng et al., 2021; Hegde …

Content truncated to 500 words