A Qualitative Evaluation Framework for Paraphrase Identification

Proceedings of Recent Advances in Natural Language Processing, pages 568–577, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_067 568 A Qualitative Evaluation Framework for Paraphrase Identification Venelin Kovatchev1,3, M. Ant`onia Mart´ı1,3, Maria Salam´o2,3, Javier Beltran1,3 1Facultat de Filolog´ıa, Universitat de Barcelona 2 Facultat de Matem`atiques i Inform`atica, Universitat de Barcelona 3Universitat de Barcelona Institute of Complex Systems Gran V´ıa de les Corts Catalanes, 585, 08007 Barcelona, Spain {vkovatchev, amarti, maria.salamo, javier.beltran}@ub.edu Abstract In this paper, we present a new approach for the evaluation, error analysis, and interpre- tation of supervised and unsupervised Para- phrase Identification (PI) systems. Our evalu- ation framework makes use of a PI corpus an- notated with linguistic phenomena to provide a better understanding and interpretation of the performance of various PI systems. Our ap- proach allows for a qualitative evaluation and comparison of the PI models using human in- terpretable categories. It does not require mod- ification of the training objective of the sys- tems and does not place additional burden on the developers. We replicate several popular supervised and unsupervised PI systems. Us- ing our evaluation framework we show that: 1) Each system performs differently with re- spect to a set of linguistic phenomena and makes qualitatively different kinds of errors; 2) Some linguistic phenomena are more chal- lenging than others across all systems. 1 Introduction In this paper we propose a new approach to eval- uation, error analysis and interpretation in the task of Paraphrase Identification (PI). Typically, PI is defined as comparing two texts of arbitrary size in order to determine whether they have approx- imately the same meaning (Dolan et al., 2004). The two texts in 1a and 1b are considered para- phrases, while the two texts at 2a and 2b are non- paraphrases.1 In 1a and 1b there is a change in the wording (“magistrate” - “judge”) and the syntac- tic structure (“was ordered” - “ordered”) but the meaning of the sentences is unchanged. In 2a and 2b there are significant differences in the quanti- ties (“5%” - “4.7%” and “$27.45” - “$27.54”). 1a A federal magistrate in Fort Lauderdale or- dered him held without bail. 1Examples are from the MRPC corpus (Dolan et al., 2004) 1b He was ordered held without bail Wednesday by a federal judge in Fort Lauderdale, Fla. 2a Microsoft fell 5 percent before the open to $27.45 from Thursday’s close of $28.91. 2b Shares in Microsoft slipped 4.7 percent in after-hours trade to $27.54 from a Nasdaq close of $28.91. The task of PI can be framed as a binary clas- sification problem. The performance of the differ- ent PI systems is reported using the Accuracy and F1 score measures. However this form of evalua- tion does not facilitate the interpretation and error analysis of the participating systems. Given the Deep Learning nature of most of the state-of-the- art systems and the complexity of the PI task, we argue that better means for evaluation, interpreta- tion, and error analysis are needed. We propose a new evaluation methodology to …

Content truncated to 500 words