Reordering Metrics for MT

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1027–1035, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Reordering Metrics for MT Alexandra Birch Miles Osborne a.birch@ed.ac.uk miles@inf.ed.ac.uk University of Edinburgh 10 Crichton Street Edinburgh, EH8 9AB, UK Abstract One of the major challenges facing statistical machine translation is how to model differ- ences in word order between languages. Al- though a great deal of research has focussed on this problem, progress is hampered by the lack of reliable metrics. Most current metrics are based on matching lexical items in the translation and the reference, and their abil- ity to measure the quality of word order has not been demonstrated. This paper presents a novel metric, the LRscore, which explic- itly measures the quality of word order by using permutation distance metrics. We show that the metric is more consistent with human judgements than other metrics, including the BLEU score. We also show that the LRscore can successfully be used as the objective func- tion when training translation model parame- ters. Training with the LRscore leads to output which is preferred by humans. Moreover, the translations incur no penalty in terms of BLEU scores. 1 Introduction Research in machine translation has focused broadly on two main goals, improving word choice and im- proving word order in translation output. Current machine translation metrics rely upon indirect meth- ods for measuring the quality of the word order, and their ability to capture the quality of word order is poor (Birch et al., 2010). There are currently two main approaches to eval- uating reordering. The first is exemplified by the BLEU score (Papineni et al., 2002), which counts the number of matching n-grams between the refer- ence and the hypothesis. Word order is captured by the proportion of longer n-grams which match. This method does not consider the position of match- ing words, and only captures ordering differences if there is an exact match between the words in the translation and the reference. Another approach is taken by two other commonly used metrics, ME- TEOR (Banerjee and Lavie, 2005) and TER (Snover et al., 2006). They both search for an alignment be- tween the translation and the reference, and from this they calculate a penalty based on the number of differences in order between the two sentences. When block moves are allowed the search space is very large, and matching stems and synonyms in- troduces errors. Importantly, none of these metrics capture the distance by which words are out of order. Also, they conflate reordering performance with the quality of the lexical items in the translation, making it difficult to tease apart the impact of changes. More sophisticated metrics, such as the RTE metric (Pad´o et al., 2009), use higher level syntactic or semantic analysis to determine the grammaticality of the out- put. These approaches require annotation and can be very slow to run. For most research, shallow metrics are more appropriate. We introduce a novel shallow …

Content truncated to 500 words