The Best Lexical Metric for Phrase-Based Statistical MT System Optimization

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 555–563, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics The Best Lexical Metric for Phrase-Based Statistical MT System Optimization Daniel Cer, Christopher D. Manning and Daniel Jurafsky Stanford University Stanford, CA 94305, USA Abstract Translation systems are generally trained to optimize BLEU, but many alternative metrics are available. We explore how optimizing toward various automatic evaluation metrics (BLEU, METEOR, NIST, TER) affects the re- sulting model. We train a state-of-the-art MT system using MERT on many parameteriza- tions of each metric and evaluate the result- ing models on the other metrics and also us- ing human judges. In accordance with popular wisdom, we find that it’s important to train on the same metric used in testing. However, we also find that training to a newer metric is only useful to the extent that the MT model’s struc- ture and features allow it to take advantage of the metric. Contrasting with TER’s good cor- relation with human judgments, we show that people tend to prefer BLEU and NIST trained models to those trained on edit distance based metrics like TER or WER. Human prefer- ences for METEOR trained models varies de- pending on the source language. Since using BLEU or NIST produces models that are more robust to evaluation by other metrics and per- form well in human judgments, we conclude they are still the best choice for training. 1 Introduction Since their introduction, automated measures of ma- chine translation quality have played a critical role in the development and evolution of SMT systems. While such metrics were initially intended for eval- uation, popular training methods such as minimum error rate training (MERT) (Och, 2003) and mar- gin infused relaxed algorithm (MIRA) (Crammer and Singer, 2003; Watanabe et al., 2007; Chiang et al., 2008) train translation models toward a specific evaluation metric. This makes the quality of the re- sulting model dependent on how accurately the au- tomatic metric actually reflects human preferences. The most popular metric for both comparing sys- tems and tuning MT models has been BLEU. While BLEU (Papineni et al., 2002) is relatively simple, scoring translations according to their n-gram over- lap with reference translations, it still achieves a rea- sonable correlation with human judgments of trans- lation quality. It is also robust enough to use for au- tomatic optimization. However, BLEU does have a number of shortcomings. It doesn’t penalize n-gram scrambling (Callison-Burch et al., 2006), and since it isn’t aware of synonymous words or phrases, it can inappropriately penalize translations that use them. Recently, there have been efforts to develop bet- ter evaluation metrics. Metrics such as Translation Edit Rate (TER) (Snover et al., 2006; Snover et al., 2009) and METEOR1 (Lavie and Denkowski, 2009) perform a more sophisticated analysis of the trans- lations being evaluated and the scores they produce tend to achieve a better correlation with human judg- ments than those produced by BLEU (Snover …

Content truncated to 500 words