AMBER: A Modified BLEU, Enhanced Ranking Metric

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 71–77, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics AMBER: A Modified BLEU, Enhanced Ranking Metric Boxing Chen and Roland Kuhn National Research Council of Canada, Gatineau, Québec, Canada First.Last@nrc.gc.ca Abstract This paper proposes a new automatic ma- chine translation evaluation metric: AMBER, which is based on the metric BLEU but incorporates recall, extra penal- ties, and some text processing variants. There is very little linguistic information in AMBER. We evaluate its system-level cor- relation and sentence-level consistency scores with human rankings from the WMT shared evaluation task; AMBER achieves state-of-the-art performance. 1 Introduction Automatic evaluation metrics for machine transla- tion (MT) quality play a critical role in the devel- opment of statistical MT systems. Several metrics have been proposed in recent years. Metrics such as BLEU (Papineni et al., 2002), NIST (Dodding- ton, 2002), WER, PER, and TER (Snover et al., 2006) do not use any linguistic information - they only apply surface matching. METEOR (Banerjee and Lavie, 2005), METEOR-NEXT (Denkowski and Lavie 2010), TER-Plus (Snover et al., 2009), MaxSim (Chan and Ng, 2008), and TESLA (Liu et al., 2010) exploit some limited linguistic resources, such as synonym dictionaries, part-of-speech tag- ging or paraphrasing tables. More sophisticated metrics such as RTE (Pado et al., 2009) and DCU- LFG (He et al., 2010) use higher level syntactic or semantic analysis to score translations. Though several of these metrics have shown bet- ter correlation with human judgment than BLEU, BLEU is still the de facto standard evaluation me- tric. This is probably due to the following facts: 1. BLEU is language independent (except for word segmentation decisions). 2. BLEU can be computed quickly. This is im- portant when choosing a metric to tune an MT system. 3. BLEU seems to be the best tuning metric from a quality point of view - i.e., models trained using BLEU obtain the highest scores from humans and even from other metrics (Cer et al., 2010). When we developed our own metric, we decided to make it a modified version of BLEU whose rankings of translations would (ideally) correlate even more highly with human rankings. Thus, our metric is called AMBER: “A Modified Bleu, En- hanced Ranking” metric. Some of the AMBER variants use an information source with a mild lin- guistic flavour – morphological knowledge about suffixes, roots and prefixes – but otherwise, the metric is based entirely on surface comparisons. 2 AMBER Like BLEU, AMBER is composed of two parts: a score and a penalty. penalty score AMBER × = (1) To address weaknesses of BLEU described in the literature (Callison-Burch et al., 2006; Lavie and Denkowski, 2009), we use more sophisticated formulae to compute the score and penalty. 2.1 Enhancing the score First, we enrich the score part with geometric av- erage of n-gram precisions (AvgP), F-measure de- rived from the arithmetic averages of precision and recall (Fmean), and arithmetic average of F- measure of precision and recall for …

Content truncated to 500 words