LRscore for Evaluating Lexical and Reordering Quality in MT

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 327–332, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics LRscore for Evaluating Lexical and Reordering Quality in MT Alexandra Birch University of Edinburgh United Kingdom a.c.birch-mayne@s0454866.ed.ac.uk Miles Osborne University of Edinburgh United Kingdom miles@inf.ed.ac.uk Abstract The ability to measure the quality of word order in translations is an important goal for research in machine translation. Cur- rent machine translation metrics do not adequately measure the reordering perfor- mance of translation systems. We present a novel metric, the LRscore, which di- rectly measures reordering success. The reordering component is balanced by a lexical metric. Capturing the two most im- portant elements of translation success in a simple combined metric with only one parameter results in an intuitive, shallow, language independent metric. 1 Introduction The main purpose of MT evaluation is to de- termine “to what extent the makers of a system have succeeded in mimicking the human transla- tor” (Krauwer, 1993). But machine translation has no “ground truth” as there are many possi- ble correct translations. It is impossible to judge whether a translation is incorrect or simply un- known and it is even harder to judge the the degree to which it is incorrect. Even so, automatic met- rics are necessary. It is nearly impossible to collect enough human judgments for evaluating incre- mental improvements in research systems, or for tuning statistical machine translation system pa- rameters. Automatic metrics are also much faster and cheaper than human evaluation and they pro- duce reproducible results. Machine translation research relies heavily upon automatic metrics to evaluate the perfor- mance of models. However, current metrics rely upon indirect methods for measuring the quality of the word order, and their ability to capture re- ordering performance has been demonstrated to be poor (Birch et al., 2010). There are two main ap- proaches to capturing reordering. The first way to measure the quality of word order is to count the number of matching n-grams between the ref- erence and the hypothesis. This is the approach taken by the BLEU score (Papineni et al., 2002). This method discounts any n-gram which is not identical to a reference n-gram, and also does not consider the relative position of the strings. They can be anywhere in the sentence. Another com- mon approach is typified by METEOR (Banerjee and Lavie, 2005) and TER (Snover et al., 2006). They calculate an ordering penalty for a hypoth- esis based on the minimum number of chunks the translation needs to be broken into in order to align it to the reference. The disadvantage of the second approach is that aligning sentences with very dif- ferent words can be inaccurate. Also there is no notion of how far these blocks are out of order. More sophisticated metrics, such as the RTE met- ric (Pad´o et al., 2009), use higher level syntactic or even semantic analysis to determine the quality of the translation. These approaches are useful, but can …

Content truncated to 500 words