Assessing the Impact of Translation Errors on Machine Translation Quality with Mixed-effects Models

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1643–1653, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Assessing the Impact of Translation Errors on Machine Translation Quality with Mixed-effects Models Marcello Federico, Matteo Negri, Luisa Bentivogli, Marco Turchi FBK - Fondazione Bruno Kessler Via Sommarive 18, 38123 Trento, Italy {federico,negri,bentivogli,turchi}@fbk.eu Abstract Learning from errors is a crucial aspect of improving expertise. Based on this no- tion, we discuss a robust statistical frame- work for analysing the impact of different error types on machine translation (MT) output quality. Our approach is based on linear mixed-effects models, which allow the analysis of error-annotated MT out- put taking into account the variability in- herent to the specific experimental setting from which the empirical observations are drawn. Our experiments are carried out on different language pairs involving Chi- nese, Arabic and Russian as target lan- guages. Interesting findings are reported, concerning the impact of different error types both at the level of human perception of quality and with respect to performance results measured with automatic metrics. 1 Introduction The dominant statistical approach to machine translation (MT) is based on learning from large amounts of parallel data and tuning the result- ing models on reference-based metrics that can be computed automatically, such as BLEU (Pap- ineni et al., 2001), METEOR (Banerjee and Lavie, 2005), TER (Snover et al., 2006), GTM (Turian et al., 2003). Despite the steady progress in the last two decades, especially for few well resourced translation directions having English as target lan- guage, this way to approach the problem is quickly reaching a performance plateau. One reason is that parallel data are a source of reliable informa- tion but, alone, limit systems knowledge to ob- served positive examples (i.e. how a sentence should be translated) without explicitly modelling any notion of error (i.e. how a sentence should not be translated). Another reason is that, as a development and evaluation criterion, automatic metrics provide a holistic view of systems’ be- haviour without identifying the specific issues of a translation. Indeed, the global scores returned by MT evaluation metrics depend on comparisons be- tween translation hypotheses and reference trans- lations, where the causes and the nature of the dif- ferences between them are not identified. To cope with these issues and define system improvement priorities, the focus of MT evalua- tion research is gradually shifting towards profil- ing systems’ behaviour with respect to various ty- pologies of errors (Vilar et al., 2006; Popovi´c and Ney, 2011; Farr´us et al., 2012, inter alia). This shift has enriched the traditional MT evaluation framework with a new element, that is the actual errors done by a system. Until now, most of the research has focused on the relationship (i.e. the correlation) between two elements of the frame- work: humans and automatic evaluation metrics. As a new element of the framework, which be- comes a sort of “evaluation triangle”, the analy- sis of error annotations opens interesting research problems related …

Content truncated to 500 words