Linguistic-based Evaluation Criteria to identify Statistical Machine Translation Errors

Linguistic-based Evaluation Criteria to identify Statistical Machine Translation Errors Mireia Farr´us*, Marta R. Costa-juss`a**, Jos´e B. Mari˜no* and Jos´e A.R. Fonollosa* *Universitat Polit`ecnica de Catalunya, TALP Research Center C/Jordi Girona 1-3, 08034 Barcelona, Spain {mfarrus,canton,adrian}@gps.tsc.upc.edu ** Barcelona Media Innovation Center Av. Diagonal 177, 08018 Barcelona, Spain marta.ruiz@barcelonamedia.org Abstract Machine translation evaluation methods are highly necessary in order to analyze the performance of translation systems. Up to now, the most traditional methods are the use of automatic measures such as BLEU or the quality perception performed by na- tive human evaluations. In order to com- plement these traditional procedures, the current paper presents a new human evalu- ation based on the expert knowledge about the errors encountered at several linguistic levels: orthographic, morphological, lexi- cal, semantic and syntactic. The results ob- tained in these experiments show that some linguistic errors could have more influence than other at the time of performing a per- ceptual evaluation. 1 Introduction One of the aims in the research community is to find accurate evaluation methods that allow ana- lyzing and comparing the performance of these translation systems. The most commonly used evaluation methods are the standard automatic measures such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover and Dorr, 2006) and WER (McCowan, 2004 et al.), as well as the use of human native evaluators that analyze and compare translated sentences according to a general perception of the linguistic quality. In this paper, these evaluation methods are used to evaluate and compare two translation systems based on the statistical approaches in the Catalan- to-Spanish language pair: Google Translate and N- c⃝ 2010 European Association for Machine Translation. II; this one developed at the Universitat Polit`ecnica de Catalunya (UPC). In addition, a new human evaluation method is applied, based on an expert linguistic evaluation, which provides information about the errors clas- sified according the level they are encountered: or- thographic, morphological, lexical, semantic and syntactic. The number of errors found in each level is then used to compare both human evalu- ations: linguistic and perceptual. Since the aim is to achieve a good human perception in our final translation, one of the main points is to see which linguistic errors have more impact in the human evaluation. The structure of this paper is as follows. Next section presents a brief summary of the related work. Section 3 presents an overview of the sta- tistical machine translation approach. Section 4 includes the description of the systems and the hu- man evaluations used in the experiments. Section 5 shows the results obtained in each of the evalua- tions, and finally, conclusions are presented in sec- tion 6. 2 Related work Automatic and human evaluation has been widely investigated by the scientific community. Having an automatic evaluation is a must in order to opti- mize a MT system. Actually, there are many inter- esting measures, for example the ones which have been presented and evaluated in the Annual Work- shop of Machine Translation (WMT) 1. Some …

Content truncated to 500 words