Sentence Level Machine Translation Evaluation as a Ranking

Proceedings of the Second Workshop on Statistical Machine Translation, pages 240–247, Prague, June 2007. c⃝2007 Association for Computational Linguistics Sentence Level Machine Translation Evaluation as a Ranking Problem: one step aside from BLEU Yang Ye University of Michigan yye@umich.edu Ming Zhou Microsoft Research Asia mingzhou@microsoft.com Chin-Yew Lin Microsoft Research Asia cyl@microsoft.com Abstract The paper proposes formulating MT evalu- ation as a ranking problem, as is often done in the practice of assessment by human. Un- der the ranking scenario, the study also in- vestigates the relative utility of several fea- tures. The results show greater correlation with human assessment at the sentence level, even when using an n-gram match score as a baseline feature. The feature contributing the most to the rank order correlation be- tween automatic ranking and human assess- ment was the dependency structure relation rather than BLEU score and reference lan- guage model feature. 1 Introduction In recent decades, alongside the growing research on Machine Translation (MT), automatic MT evalu- ation has become a critical problem for MT system developers, who are interested in quick turnaround development cycles. The state-of-the-art automatic MT evaluation is an n-gram based metric repre- sented by BLEU (Papineni et al., 2001) and its vari- ants. Ever since its creation, the BLEU score has been the gauge of Machine Translation system eval- uation. Nevertheless, the research community has been largely aware of the deficiency of the BLEU metric. BLEU captures only a single dimension of the vitality of natural languages: a candidate translation gets acknowledged only if it uses ex- actly the same lexicon as the reference translation. Natural languages, however, are characterized by their extremely rich mechanisms for reproduction via a large number of syntactic, lexical and semantic rewriting rules. Although BLEU has been shown to correlate positively with human assessments at the document level (Papineni et al., 2001), efforts to improve state-of-the-art MT require that human as- sessment be approximated at sentence level as well. Researchers report the BLEU score at document level in order to combat the sparseness of n-grams in BLEU scoring. But, ultimately, document-level MT evaluation has to be pinned down to the gran- ularity of the sentence. Unfortunately, the corre- lation between human assessment and BLEU score at sentence level is extremely low (Liu et al., 2005, 2006). While acknowledging the appealing simplic- ity of BLEU as a way to access one perspective of an MT candidate translation’s quality, we observe the following facts of n-gram based MT metrics. First, they may not reflect the mechanism of how human beings evaluate sentence translation quality. More specifically, optimizing BLEU does not guarantee the optimization of sentence quality approved by hu- man assessors. Therefore, BLEU is likely to have a low correlation with human assessment at sen- tence level for most candidate translations. Second, it is conceivable that human beings are more reli- able ranking the quality of multiple candidate trans- lations than assigning a numeric value to index the quality of the candidate translation even with signif- icant …

Content truncated to 500 words