Class error rates for evaluation of machine translation output

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 71–75, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Class error rates for evaluation of machine translation output Maja Popovi´c German Research Center for Artificial Intelligence (DFKI) Language Technology (LT), Berlin, Germany maja.popovic@dfki.de Abstract We investigate the use of error classification results for automatic evaluation of machine translation output. Five basic error classes are taken into account: morphological errors, syn- tactic (reordering) errors, missing words, ex- tra words and lexical errors. In addition, lin- ear combinations of these categories are in- vestigated. Correlations between the class er- ror rates and human judgments are calculated on the data of the third, fourth, fifth and sixth shared tasks of the Statistical Machine Trans- lation Workshop. Machine translation outputs in five different European languages are used: English, Spanish, French, German and Czech. The results show that the following combina- tions are the most promising: the sum of all class error rates, the weighted sum optimised for translation into English and the weighted sum optimised for translation from English. 1 Introduction Recent investigations have shown that it is possi- ble to carry out a reliable automatic error analysis of a given translation output in order to get more information about actual errors and details about particular strengthnesses and weaknesses of a sys- teml (Popovi´c and Ney, 2011). The obtained results correlate very well with the human error classifica- tion results. The question we try to answer is: how the class error rates correlate with the human eval- uation (ranking) results? As a first step, we inves- tigate the correlations of five basic class error rates with human rankings. In the next step, linear com- binations (sums) of basic class error rates are inves- tigated. Spearman’s rank correlation coefficients on the document (system) level between all the metrics and the human ranking are computed on the English, French, Spanish, German and Czech texts gener- ated by various translation systems in the frame- work of the third (Callison-Burch et al., 2008), fourth (Callison-Burch et al., 2009), fifth (Callison- Burch et al., 2010) and sixth (Callison-Burch et al., 2011) shared translation tasks. 2 Class error rates In this work, the method proposed in (Popovi´c and Ney, 2011) is used, i.e. classification of the translation errors into five basic categories based on the Word Error Rate (WER) (Levenshtein, 1966) together with the recall- and precision-based Position-independent Error Rates called Reference PER (RPER) and Hypothesis PER (HPER). As a result of an error classification, two values are usually of interest: raw error counts for each er- ror class, and error rates for each class, i.e. raw error counts normalised over the total number of running words. Which of the values is preferred depends of the exact task. For example, if only a distribution of error types within a translation output is of in- terest, the raw error counts are sufficient. On the other hand, if we want to compare different transla- tion outputs, normalised values i.e. error …

Content truncated to 500 words