A Lightweight Evaluation Framework for Machine Translation Reordering

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 12–21, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics A Lightweight Evaluation Framework for Machine Translation Reordering David Talbot1 and Hideto Kazawa2 and Hiroshi Ichikawa2 Jason Katz-Brown2 and Masakazu Seno2 and Franz J. Och1 1 Google Inc. 2 Google Japan 1600 Amphitheatre Parkway Roppongi Hills Mori Tower Mountain View, CA 94043 6-10-1 Roppongi, Tokyo 106-6126 {talbot, och}@google.com {kazawa, ichikawa}@google.com {jasonkb, seno}@google.com Abstract Reordering is a major challenge for machine translation between distant languages. Recent work has shown that evaluation metrics that explicitly account for target language word or- der correlate better with human judgments of translation quality. Here we present a simple framework for evaluating word order indepen- dently of lexical choice by comparing the sys- tem’s reordering of a source sentence to ref- erence reordering data generated from manu- ally word-aligned translations. When used to evaluate a system that performs reordering as a preprocessing step our framework allows the parser and reordering rules to be evaluated ex- tremely quickly without time-consuming end- to-end machine translation experiments. novelty of our approach is that the translationsA used to generate the reordering reference data are generated in an alignment-oriented fash- ion. We show that how the alignments are generated can significantly effect the robust- ness of the evaluation. We also outline some ways in which this framework has allowed our group to analyze reordering errors for English to Japanese machine translation. 1 Introduction Statistical machine translation systems can perform poorly on distant language pairs such as English and Japanese. Reordering errors are a major source of poor or misleading translations in such systems (Isozaki et al., 2010). Unfortunately the stan- dard evaluation metrics used by the statistical ma- chine translation community are relatively insensi- tive to the long-distance reordering phenomena en- countered when translating between such languages (Birch et al., 2010). The ability to rapidly evaluate the impact of changes on a system can significantly accelerate the experimental cycle. In a large statistical machine translation system, we should ideally be able to ex- periment with separate components without retrain- ing the complete system. Measures such as per- plexity have been successfully used to evaluate lan- guage models independently in speech recognition eliminating some of the need for end-to-end speech recognition experiments. In machine translation, alignment error rate has been used with some mixed success to evaluate word-alignment algorithms but no standard evaluation frameworks exist for other components of a machine translation system (Fraser and Marcu, 2007). Unfortunately, BLEU (Papineni et al., 2001) and other metrics that work with the final output of a ma- chine translation system are both insensitive to re- ordering phenomena and relatively time-consuming to compute: changes to the system may require the realignment of the parallel training data, extraction of phrasal statistics and translation of a test set. As training sets grow in size, the cost of end-to-end ex- perimentation can become significant. However, it is not clear that measurements made on any single part …

Content truncated to 500 words