ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation

ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation Chin-Yew Lin and Franz Josef Och Information Sciences Institute University of Southern California 4676 Admiralty Way Marina del Rey, CA 90292, USA {cyl,och}@isi.edu Abstract Comparisons of automatic evaluation metrics for machine translation are usually conducted on corpus level using correlation statistics such as Pearson’s product moment correlation coefficient or Spearman’s rank order correlation coefficient between human scores and automatic scores. However, such comparisons rely on human judgments of translation qualities such as adequacy and fluency. Unfortunately, these judgments are often inconsistent and very expensive to acquire. In this paper, we introduce a new evaluation method, ORANGE, for evaluating automatic machine translation evaluation metrics automatically without extra human involvement other than using a set of reference translations. We also show the results of comparing several existing automatic metrics and three new automatic metrics using ORANGE. 1 Introduction To automatically evaluate machine translations, the machine translation community recently adopted an n-gram co-occurrence scoring procedure BLEU (Papineni et al. 2001). A similar metric, NIST, used by NIST (NIST 2002) in a couple of machine translation evaluations in the past two years is based on BLEU. The main idea of BLEU is to measure the translation closeness between a candidate translation and a set of reference translations with a numerical metric. Although the idea of using objective functions to automatically evaluate machine translation quality is not new (Su et al. 1992), the success of BLEU prompts a lot of interests in developing better automatic evaluation metrics. For example, Akiba et al. (2001) proposed a metric called RED based on edit distances over a set of multiple references. Nießen et al. (2000) calculated the length- normalized edit distance, called word error rate (WER), between a candidate and multiple reference translations. Leusch et al. (2003) proposed a related measure called position- independent word error rate (PER) that did not consider word position, i.e. using bag-of-words instead. Turian et al. (2003) introduced General Text Matcher (GTM) based on accuracy measures such as recall, precision, and F-measure. With so many different automatic metrics available, it is necessary to have a common and objective way to evaluate these metrics. Comparison of automatic evaluation metrics are usually conducted on corpus level using correlation analysis between human scores and automatic scores such as BLEU, NIST, WER, and PER. However, the performance of automatic metrics in terms of human vs. system correlation analysis is not stable across different evaluation settings. For example, Table 1 shows the Pearson’s linear correlation coefficient analysis of 8 machine translation systems from 2003 NIST Chinese- English machine translation evaluation. The Pearson’ correlation coefficients are computed according to different automatic evaluation methods vs. human assigned adequacy and fluency. BLEU1, 4, and 12 are BLEU with maximum n-gram lengths of 1, 4, and 12 respectively. GTM10, 20, and 30 are GTM with exponents of 1.0, 2.0, and 3.0 respectively. 95% confidence intervals are estimated using bootstrap resampling (Davison and Hinkley 1997). From the BLEU group, we found that …

Content truncated to 500 words