Re-evaluating Machine Translation Results with Paraphrase Support

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 77–84, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Re-evaluating Machine Translation Results with Paraphrase Support Liang Zhou, Chin-Yew Lin, and Eduard Hovy University of Southern California Information Sciences Institute 4676 Admiralty Way Marina del Rey, CA 90292-6695 {liangz, cyl, hovy} @isi.edu Abstract In this paper, we present ParaEval, an automatic evaluation framework that uses paraphrases to improve the quality of machine translation evaluations. Previous work has focused on fixed n-gram evaluation metrics coupled with lexical identity matching. ParaEval addresses three important issues: support for para- phrase/synonym matching, recall meas- urement, and correlation with human judgments. We show that ParaEval corre- lates significantly better than BLEU with human assessment in measurements for both fluency and adequacy. 1 Introduction The introduction of automated evaluation proce- dures, such as BLEU (Papineni et al., 2001) for machine translation (MT) and ROUGE (Lin and Hovy, 2003) for summarization, have prompted much progress and development in both of these areas of research in Natural Language Processing (NLP). Both evaluation tasks employ a compari- son strategy for comparing textual units from machine-generated and gold-standard texts. Ide- ally, this comparison process would be per- formed manually, because of humans’ abilities to infer, paraphrase, and use world knowledge to relate differently worded pieces of equivalent information. However, manual evaluations are time consuming and expensive, thus making them a bottleneck in system development cycles. BLEU measures how close machine-generated translations are to professional human transla- tions, and ROUGE does the same with respect to summaries. Both methods incorporate the com- parison of a system-produced text to one or more corresponding reference texts. The closeness be- tween texts is measured by the computation of a numeric score based on n-gram co-occurrence statistics. Although both methods have gained mainstream acceptance and have shown good correlations with human judgments, their defi- ciencies have become more evident and serious as research in MT and summarization progresses (Callison-Burch et al., 2006). Text comparisons in MT and summarization evaluations are performed at different text granu- larity levels. Since most of the phrase-based, syntax-based, and rule-based MT systems trans- late one sentence at a time, the text comparison in the evaluation process is also performed at the single-sentence level. In summarization evalua- tions, there is no sentence-to-sentence corre- spondence between summary pairs—essentially a multi-sentence-to-multi-sentence comparison, making it more difficult and requiring a com- pletely different implementation for matching strategies. In this paper, we focus on the intrica- cies involved in evaluating MT results and ad- dress two prominent problems associated with the BLEU-esque metrics, namely their lack of support for paraphrase matching and the absence of recall scoring. Our solution, ParaEval, utilizes a large collection of paraphrases acquired through an unsupervised process—identifying phrase sets that have the same translation in an- other language—using state-of-the-art statistical MT word alignment and phrase extraction meth- ods. This collection facilitates paraphrase match- ing, additionally coupled with lexical identity matching which is designed for comparing text/sentence …

Content truncated to 500 words