From Human to Automatic Error Classification for Machine Translation Output

From Human to Automatic Error Classification for Machine Translation Output Maja Popovi´c and Aljoscha Burchardt German Research Center for Artificial Intelligence (DFKI) Language Technology Group (LT) Berlin, Germany {maja.popovic,aljoscha.burchardt}@dfki.de Abstract Future improvement of machine transla- tion systems requires reliable automatic evaluation and error classification mea- sures to avoid time and money consuming human classification. In this article, we propose a new method for automatic er- ror classification and systematically com- pare its results to those obtained by hu- mans. We show that the proposed auto- matic measures correlate well with human judgments across different error classes as well as across different translation outputs on four out of five commonly used error classes. 1 Introduction and related work The evaluation of machine translation output is an intrinsically difficult task. Human evaluation is expensive and time consuming. Therefore a great deal of effort has been spent on finding mea- sures that correlate well with human judgements when ranking translation systems for quality (see for example (Callison-Burch et al., 2009; Callison- Burch et al., 2010)). A considerable amount of work has been put into the improvement of these measures. However, most of the work has been fo- cused just on ranking between different machine translation systems. While ranking systems is an important first step towards their improvement, it does not provide enough scientific insights. Re- searchers often would find it helpful to get an- swers to questions such as What is a particular strength/weakness of my system? What kind of errors does the system make most often? Does a particular modification improve some aspect of c⃝ 2011 European Association for Machine Translation. the system, even if it does not improve the overall score? Does a worse–ranked system outperform a better–ranked one in any aspect?, etc. In order to answer such questions, a framework for human error analysis and error classification has been proposed in (Vilar et al., 2006), where a classification scheme based on (Llitj´os et al., 2005) is presented together with a detailed analysis of the obtained results. The method has become widely used in recent years (Avramidis and Koehn, 2008; Max et al., 2008; Khalilov and Fonollosa, 2009; Li et al., 2009). Still, human error classification is resource-intensive and might become practically unfeasible when translating into many languages. As for automatic methods, an approach for au- tomatic identification of patterns in translation out- put using POS sequences is proposed in (Lopez and Resnik, 2005) in order to see how well a translation system is capable of capturing system- atic reordering patterns. Using relative differences between Word Error Rate (WER) and Position- independent Word Error Rate (PER) for nouns, ad- jectives and verbs has been proposed in (Popovi´c et al., 2006) for the estimation of inflectional and reordering errors. A method based on WER and PER decomposition for discovering inflectional errors and missing words is presented in (Popovi´c and Ney, 2007). Zhou (2008) proposed a diagnos- tic evaluation of linguistic check-points obtained automatically by aligning parsed source and target sentences. However, to …

Content truncated to 500 words