Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output

Proceedings of the Workshop on Statistical Machine Translation, pages 1–6, New York City, June 2006. c⃝2006 Association for Computational Linguistics Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output Maja Popovi´c⋆ Hermann Ney⋆ Adri`a de Gispert† Jos´e B. Mari˜no† Deepa Gupta⊥ Marcello Federico⊥ Patrik Lambert† Rafael Banchs† ⋆ Lehrstuhl f¨ur Informatik VI - Computer Science Department, RWTH Aachen University, Aachen, Germany † TALP Research Center, Universitat Polit`ecnica de Catalunya (UPC), Barcelona, Spain ⊥ ITC-irst, Centro per la Ricerca Scientifica e Tecnologica, Trento, Italy {popovic,ney}@informatik.rwth-aachen.de {agispert,canton}@gps.tsc.upc.es {gupta,federico}@itc.it {lambert,banchs}@gps.tsc.upc.es Abstract Evaluation of machine translation output is an important but difficult task. Over the last years, a variety of automatic evalua- tion measures have been studied, some of them like Word Error Rate (WER), Posi- tion Independent Word Error Rate (PER) and BLEU and NIST scores have become widely used tools for comparing different systems as well as for evaluating improve- ments within one system. However, these measures do not give any details about the nature of translation errors. Therefore some analysis of the generated output is needed in order to identify the main prob- lems and to focus the research efforts. On the other hand, human evaluation is a time consuming and expensive task. In this paper, we investigate methods for using of morpho-syntactic information for auto- matic evaluation: standard error measures WER and PER are calculated on distinct word classes and forms in order to get a better idea about the nature of translation errors and possibilities for improvements. 1 Introduction The evaluation of the generated output is an impor- tant issue for all natural language processing (NLP) tasks, especially for machine translation (MT). Au- tomatic evaluation is preferred because human eval- uation is a time consuming and expensive task. A variety of automatic evaluation measures have been proposed and studied over the last years, some of them are shown to be a very useful tool for com- paring different systems as well as for evaluating improvements within one system. The most widely used are Word Error Rate (WER), Position Indepen- dent Word Error Rate (PER), the BLEU score (Pap- ineni et al., 2002) and the NIST score (Doddington, 2002). However, none of these measures give any details about the nature of translation errors. A rela- tionship between these error measures and the actual errors in the translation outputs is not easy to find. Therefore some analysis of the translation errors is necessary in order to define the main problems and to focus the research efforts. A framework for hu- man error analysis and error classification has been proposed in (Vilar et al., 2006), but like human eval- uation, this is also a time consuming task. The goal of this work is to present a framework for automatic error analysis of machine translation output based on morpho-syntactic information. 2 Related Work There is a number of publications dealing with various automatic evaluation measures for machine translation output, some of them proposing new measures, some proposing improvements and exten- sions …

Content truncated to 500 words