PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 923–932, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts Chang Liu1 and Daniel Dahlmeier2 and Hwee Tou Ng1,2 1Department of Computer Science, National University of Singapore 2NUS Graduate School for Integrative Sciences and Engineering {liuchan1,danielhe,nght}@comp.nus.edu.sg Abstract We present PEM, the first fully automatic met- ric to evaluate the quality of paraphrases, and consequently, that of paraphrase generation systems. Our metric is based on three crite- ria: adequacy, fluency, and lexical dissimilar- ity. The key component in our metric is a ro- bust and shallow semantic similarity measure based on pivot language N-grams that allows us to approximate adequacy independently of lexical similarity. Human evaluation shows that PEM achieves high correlation with hu- man judgments. 1 Introduction In recent years, there has been an increasing inter- est in the task of paraphrase generation (PG) (Barzi- lay and Lee, 2003; Pang et al., 2003; Quirk et al., 2004; Bannard and Callison-Burch, 2005; Kauchak and Barzilay, 2006; Zhao et al., 2008; Zhao et al., 2009). At the same time, the task has seen appli- cations such as machine translation (MT) (Callison- Burch et al., 2006; Madnani et al., 2007; Madnani et al., 2008), MT evaluation (Kauchak and Barzilay, 2006; Zhou et al., 2006a; Owczarzak et al., 2006), summary evaluation (Zhou et al., 2006b), and ques- tion answering (Duboue and Chu-Carroll, 2006). Despite the research activities, we see two major problems in the field. First, there is currently no con- sensus on what attributes characterize a good para- phrase. As a result, works on the application of para- phrases tend to build their own PG system in view of the immediate needs instead of using an existing system. Second, and as a consequence, no automatic eval- uation metric exists for paraphrases. Most works in this area resort to ad hoc manual evaluations, such as the percentage of “yes” judgments to the question of “is the meaning preserved”. This type of evaluation is incomprehensive, expensive, and non-comparable between different studies, making progress hard to judge. In this work we address both problems. We pro- pose a set of three criteria for good paraphrases: ad- equacy, fluency, and lexical dissimilarity. Consid- ering that paraphrase evaluation is a very subjec- tive task with no rigid definition, we conduct ex- periments with human judges to show that humans generally have a consistent intuition for good para- phrases, and that the three criteria are good indica- tors. Based on these criteria, we construct PEM (Para- phrase Evaluation Metric), a fully automatic evalua- tion metric for PG systems. PEM takes as input the original sentence R and its paraphrase candidate P, and outputs a single numeric score b estimating the quality of P as a paraphrase of R. PG systems can be compared based on the average scores of their output paraphrases. To the best of our knowledge, this is the first automatic metric that gives …

Content truncated to 500 words