A Simple Automatic MT Evaluation Metric

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 33–36, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics A Simple Automatic MT Evaluation Metric Petr Homola Charles University Prague, Czech Republic Vladislav Kuboˇn Charles University Prague, Czech Republic {homola|vk|pecina}@ufal.mff.cuni.cz Pavel Pecina Charles University Prague, Czech Republic Abstract This paper describes a simple evaluation metric for MT which attempts to overcome the well-known deficits of the standard BLEU metric from a slightly different an- gle. It employes Levenshtein’s edit dis- tance for establishing alignment between the MT output and the reference transla- tion in order to reflect the morphological properties of highly inflected languages. It also incorporates a very simple measure expressing the differences in the word or- der. The paper also includes evaluation on the data from the previous SMT workshop for several language pairs. 1 Introduction The problem of finding a reliable machine trans- lation metrics corresponding with a human judg- ment has recently returned to the centre of atten- tion. After a brief period following the introduc- tion of generally accepted and widely used met- rics, BLEU (Papineni et al., 2002) and NIST (Dod- dington, 2002), when it seemed that this persistent problem has finally been solved, the researchers active in the field of machine translation (MT) started to express their worries that although these metrics are simple, fast and able to provide con- sistent results for a particular system during its de- velopment, they are not sufficiently reliable for the comparison of different systems or different lan- guage pairs. The results of the NIST evaluation in 2005 (Le and Przybocki, 2005) have also strengthened the suspicion that the correlation between human judgment and the BLEU and NIST measures is not as strong as it was widely believed. Both mea- sures seem to favor the MT output created by sys- tems based on n-gram architecture, they are un- able to take into account certain factors which are very important for the human judges of translation quality. The article (Callison-Burch et al., 2006) thor- oughly discusses the deficits of the BLEU and similar metrics. The authors claim that the existing automatic metrics, including some of the new and seemingly more reliable ones as e.g. Meteor (cf. (Banerjee and Lavie, 2005)) ”...they are all quite rough measures of translation similarity, and have inexact models of allowable variation in transla- tion.” This claim is supported by a construction of translation variations which have identical BLEU score, but which are very different for a human judge. The authors identify three prominent fac- tors which contribute to the inadequacy of BLEU – the failure to deal with synonyms and paraphrases, no penalties for missing content, and the crudeness of the brevity penalty. Let us add some more factors based on our ex- periments with languages typologically different than English, Arabic or Chinese, which are prob- ably the languages most frequently used in recent shared-task MT evaluations. The highly inflected languages and languages with a higher degree …

Content truncated to 500 words