Re-examining Machine Translation Metrics for Paraphrase Identification

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 182–190, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Re-examining Machine Translation Metrics for Paraphrase Identification Nitin Madnani Joel Tetreault Educational Testing Service Princeton, NJ, USA {nmadnani,jtetreault}@ets.org Martin Chodorow Hunter College of CUNY New York, NY, USA martin.chodorow@hunter.cuny.edu Abstract We propose to re-examine the hypothesis that automated metrics developed for MT evalu- ation can prove useful for paraphrase iden- tification in light of the significant work on the development of new MT metrics over the last 4 years. We show that a meta-classifier trained using nothing but recent MT metrics outperforms all previous paraphrase identifi- cation approaches on the Microsoft Research Paraphrase corpus. In addition, we apply our system to a second corpus developed for the task of plagiarism detection and obtain ex- tremely positive results. Finally, we conduct extensive error analysis and uncover the top systematic sources of error for a paraphrase identification approach relying solely on MT metrics. We release both the new dataset and the error analysis annotations for use by the community. 1 Introduction One of the most important reasons for the recent advances made in Statistical Machine Translation (SMT) has been the development of automated met- rics for evaluation of translation quality. The goal of any such metric is to assess whether the trans- lation hypothesis produced by a system is seman- tically equivalent to the source sentence that was translated. However, cross-lingual semantic equiv- alence is even harder to assess than monolingual, therefore, most MT metrics instead try to measure whether the hypothesis is semantically equivalent to a human-authored reference translation of the same source sentence. Using such automated metrics as proxies for human judgments can provide a quick as- sessment of system performance and allow for short feature and system development cycles, which are important for evaluating research ideas. In the last 5 years, several shared tasks and com- petitions have led to the development of increasingly sophisticated metrics that go beyond the computa- tion of n-gram overlaps (BLEU, NIST) or edit dis- tances (TER, WER, PER etc.). Note that the task of an MT metric is essentially one of identifying whether the translation produced by a system is a paraphrase of the reference translation. Although the notion of using MT metrics for the task of para- phrase identification is not novel (Finch et al., 2005; Wan et al., 2006), it merits a re-examination in the light of the development of these novel MT metrics for which we can ask “How much better, if at all, do these newer metrics perform for the task of para- phrase identification?” This paper describes such a re-examination. We employ 8 different MT metrics for identifying paraphrases across two different datasets - the well-known Microsoft Research paraphrase corpus (MSRP) (Dolan et al., 2004) and the plagiarism detection corpus (PAN) from the 2010 Uncovering Plagiarism, Authorship and Social Software Misuse shared task (Potthast et al., 2010). We include both …

Content truncated to 500 words