Improving AMBER, an MT Evaluation Metric

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 59–63, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Improving AMBER, an MT Evaluation Metric Boxing Chen, Roland Kuhn and George Foster National Research Council Canada 283 Alexandre-Taché Boulevard, Gatineau (Québec), Canada J8X 3X7 {Boxing.Chen, Roland.Kuhn, George.Foster}@nrc.ca Abstract A recent paper described a new machine translation evaluation metric, AMBER. This paper describes two changes to AMBER. The first one is incorporation of a new ordering penalty; the second one is the use of the downhill simplex algorithm to tune the weights for the components of AMBER. We tested the impact of the two changes, using data from the WMT metrics task. Each of the changes by itself improved the performance of AMBER, and the two together yielded even greater improvement, which in some cases was more than additive. The new version of AMBER clearly outperforms BLEU in terms of correlation with human judgment. 1 Introduction AMBER is a machine translation evaluation metric first described in (Chen and Kuhn, 2011). It is de- signed to have the advantages of BLEU (Papineni et al., 2002), such as nearly complete language independence and rapid computability, while at- taining even higher correlation with human judg- ment. According to the paper just cited: “It can be thought of as a weighted combination of dozens of computationally cheap features based on word sur- face forms for evaluating MT quality”. Many re- cently defined machine translation metrics seek to exploit deeper sources of knowledge than are available to BLEU, such as external lexical and syntactic resources. Unlike these and like BLEU, AMBER relies entirely on matching surface forms in tokens in the hypothesis and reference, thus sac- rificing depth of knowledge for simplicity and speed. In this paper, we describe two improvements to AMBER. The first is a new ordering penalty called “v” developed in (Chen et al., 2012). The second remedies a weakness in the 2011 version of AMBER by carrying out automatic, rather than manual, tuning of this metric’s free parameters; we now use the simplex algorithm to do the tuning. 2 AMBER AMBER is the product of a score and a penalty, as in Equation (1); in this, it resembles BLEU. How- ever, both the score part and the penalty part are more sophisticated than in BLEU. The score part (Equation 2) is enriched by incorporating the weighted average of n-gram precisions (AvgP), the F-measure derived from the arithmetic averages of precision and recall (Fmean), and the arithmetic average of F-measure of precision and recall for each n-gram (AvgF). The penalty part is a weighted product of several different penalties (Equation 3). Our original AMBER paper (Chen and Kuhn, 2011) describes the ten penalties used at that time; two of these penalties, the normalized Spearman’s correlation penalty and the normalized Kendall’s correlation penalty, model word reorder- ing. penalty score AMBER × = (1) AvgF Fmean AvgP score × − − + × + × = ) 1( 2 1 2 1 θ …

Content truncated to 500 words