Word Error Rates: Decomposition over POS classes and Applications for Error Analysis

Proceedings of the Second Workshop on Statistical Machine Translation, pages 48–55, Prague, June 2007. c⃝2007 Association for Computational Linguistics Word Error Rates: Decomposition over POS Classes and Applications for Error Analysis Maja Popovi´c Lehrstuhl f¨ur Informatik 6 RWTH Aachen University Aachen, Germany popovic@cs.rwth-aachen.de Hermann Ney Lehrstuhl f¨ur Informatik 6 RWTH Aachen University Aachen, Germany ney@cs.rwth-aachen.de Abstract Evaluation and error analysis of machine translation output are important but difficult tasks. In this work, we propose a novel method for obtaining more details about ac- tual translation errors in the generated output by introducing the decomposition of Word Error Rate (WER) and Position independent word Error Rate (PER) over different Part- of-Speech (POS) classes. Furthermore, we investigate two possible aspects of the use of these decompositions for automatic er- ror analysis: estimation of inflectional errors and distribution of missing words over POS classes. The obtained results are shown to correspond to the results of a human error analysis. The results obtained on the Euro- pean Parliament Plenary Session corpus in Spanish and English give a better overview of the nature of translation errors as well as ideas of where to put efforts for possible im- provements of the translation system. 1 Introduction Evaluation of machine translation output is a very important but difficult task. Human evaluation is expensive and time consuming. Therefore a variety of automatic evaluation measures have been studied over the last years. The most widely used are Word Error Rate (WER), Position independent word Error Rate (PER), the BLEU score (Papineni et al., 2002) and the NIST score (Doddington, 2002). These mea- sures have shown to be valuable tools for comparing different systems as well as for evaluating improve- ments within one system. However, these measures do not give any details about the nature of translation errors. Therefore some more detailed analysis of the generated output is needed in order to identify the main problems and to focus the research efforts. A framework for human error analysis has been pro- posed in (Vilar et al., 2006), but as every human evaluation, this is also a time consuming task. This article presents a framework for calculating the decomposition of WER and PER over different POS classes, i.e. for estimating the contribution of each POS class to the overall word error rate. Al- though this work focuses on POS classes, the method can be easily extended to other types of linguis- tic information. In addition, two methods for error analysis using the WER and PER decompositons to- gether with base forms are proposed: estimation of inflectional errors and distribution of missing words over POS classes. The translation corpus used for our error analysis is built in the framework of the TC-STAR project (tcs, 2005) and contains the tran- scriptions of the European Parliament Plenary Ses- sions (EPPS) in Spanish and English. The translation system used is the phrase-based statistical machine translation system described in (Vilar et al., 2005; Matusov et al., 2006). 2 Related Work Automatic evaluation measures for machine trans- …

Content truncated to 500 words