CDER: Efficient MT Evaluation Using Block Movements

CDER: Efficient MT Evaluation Using Block Movements Gregor Leusch and Nicola Ueffing and Hermann Ney Lehrstuhl f¨ur Informatik VI, Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {leusch,ueffing,ney}@i6.informatik.rwth-aachen.de Abstract Most state-of-the-art evaluation measures for machine translation assign high costs to movements of word blocks. In many cases though such movements still result in correct or almost correct sentences. In this paper, we will present a new eval- uation measure which explicitly models block reordering as an edit operation. Our measure can be exactly calculated in quadratic time. Furthermore, we will show how some evaluation measures can be improved by the introduction of word-dependent substitution costs. The correlation of the new measure with human judgment has been investigated systematically on two different language pairs. The experimental results will show that it significantly outperforms state-of-the-art approaches in sentence-level correlation. Results from experiments with word dependent substi- tution costs will demonstrate an additional increase of correlation between automatic evaluation measures and human judgment. 1 Introduction Research in machine translation (MT) depends heavily on the evaluation of its results. Espe- cially for the development of an MT system, an evaluation measure is needed which reliably assesses the quality of MT output. Such a measure will help analyze the strengths and weaknesses of different translation systems or different versions of the same system by comparing output at the sentence level. In most applications of MT, understandability for humans in terms of readability as well as semantical correctness should be the evaluation criterion. But as human evaluation is tedious and cost-intensive, automatic evaluation measures are used in most MT research tasks. A high correlation between these automatic evaluation measures and human evaluation is thus desirable. State-of-the-art measures such as BLEU (Pap- ineni et al., 2002) or NIST (Doddington, 2002) aim at measuring the translation quality rather on the document level1 than on the level of single sentences. They are thus not well-suited for sentence-level evaluation. The introduction of smoothing (Lin and Och, 2004) solves this problem only partially. In this paper, we will present a new automatic error measure for MT – the CDER – which is designed for assessing MT quality on the sentence level. It is based on edit distance – such as the well-known word error rate (WER) – but allows for reordering of blocks. Nevertheless, by defining reordering costs, the ordering of the words in a sentence is still relevant for the measure. In this, the new measure differs significantly from the position independent error rate (PER) by (Tillmann et al., 1997). Generally, finding an optimal solution for such a reordering problem is NP hard, as is shown in (Lopresti and Tomkins, 1997). In previous work, researchers have tried to reduce the complexity, for example by restricting the possible permutations on the block-level, or by approximation or heuristics during the calculation. Nevertheless, most of the resulting algorithms still have high run times and are hardly applied in practice, or give only a rough approximation. An overview of some better-known measures can …

Content truncated to 500 words