Paraphrasing for Automatic Evaluation

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 455–462, New York, June 2006. c⃝2006 Association for Computational Linguistics Paraphrasing for Automatic Evaluation David Kauchak Department of Computer Science University of California, San Diego dkauchak@cs.ucsd.edu Regina Barzilay CSAIL Massachusetts Institute of Technology regina@csail.mit.edu Abstract This paper studies the impact of para- phrases on the accuracy of automatic eval- uation. Given a reference sentence and a machine-generated sentence, we seek to find a paraphrase of the reference sen- tence that is closer in wording to the ma- chine output than the original reference. We apply our paraphrasing method in the context of machine translation evaluation. Our experiments show that the use of a paraphrased synthetic reference refines the accuracy of automatic evaluation. We also found a strong connection between the quality of automatic paraphrases as judged by humans and their contribution to automatic evaluation. 1 Introduction The use of automatic methods for evaluating machine-generated text is quickly becoming main- stream in natural language processing. The most notable examples in this category include measures such as BLEU and ROUGE which drive research in the machine translation and text summarization communities. These methods assess the quality of a machine-generated output by considering its simi- larity to a reference text written by a human. Ideally, the similarity would reflect the semantic proximity between the two. In practice, this comparison breaks down to n-gram overlap between the reference and the machine output. 1a. However, Israel’s reply failed to completely clear the U.S. suspicions. 1b. However, Israeli answer unable to fully remove the doubts. Table 1: A reference sentence and corresponding machine translation from the NIST 2004 MT eval- uation. Consider the human-written translation and the machine translation of the same Chinese sentence shown in Table 1. While the two translations con- vey the same meaning, they share only auxiliary words. Clearly, any measure based on word over- lap will penalize a system for generating such a sen- tence. The question is whether such cases are com- mon phenomena or infrequent exceptions. Empiri- cal evidence supports the former. Analyzing 10,728 reference translation pairs1 used in the NIST 2004 machine translation evaluation, we found that only 21 (less than 0.2%) of them are identical. Moreover, 60% of the pairs differ in at least 11 words. These statistics suggest that without accounting for para- phrases, automatic evaluation measures may never reach the accuracy of human evaluation. As a solution to this problem, researchers use multiple references to refine automatic evaluation. Papineni et al. (2002) shows that expanding the number of references reduces the gap between au- tomatic and human evaluation. However, very few human annotated sets are augmented with multiple references and those that are available are relatively 1Each pair included different translations of the same sen- tence, produced by two human translators. 455 small in size. Moreover, access to several references does not guarantee that the references will include the same words that appear in machine-generated sentences. In this paper, …

Content truncated to 500 words