Improving Evaluation of Machine Translation Quality Estimation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1804–1813, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Improving Evaluation of Machine Translation Quality Estimation Yvette Graham ADAPT Centre School of Computer Science and Statistics Trinity College Dublin yvette.graham@scss.tcd.ie Abstract Quality estimation evaluation commonly takes the form of measurement of the error that exists between predictions and gold standard labels for a particular test set of translations. Issues can arise during com- parison of quality estimation prediction score distributions and gold label distribu- tions, however. In this paper, we provide an analysis of methods of comparison and identify areas of concern with respect to widely used measures, such as the ability to gain by prediction of aggregate statistics specific to gold label distributions or by optimally conservative variance in predic- tion score distributions. As an alternative, we propose the use of the unit-free Pear- son correlation, in addition to providing an appropriate method of significance testing improvements over a baseline. Compo- nents of WMT-13 and WMT-14 quality es- timation shared tasks are replicated to re- veal substantially increased conclusivity in system rankings, including identification of outright winners of tasks. 1 Introduction Machine Translation (MT) Quality Estimation (QE) is the automatic prediction of machine trans- lation quality without the use of reference trans- lations (Blatz et al., 2004; Specia et al., 2009). Human assessment of translation quality in theory provides the most meaningful evaluation of sys- tems, but human assessors are known to be incon- sistent and this causes challenges for quality es- timation evaluation. For instance, there is a gen- eral lack of consensus both with respect to what provides the most meaningful gold standard rep- resentation, as well as best method of compari- son of gold labels and system predictions. For ex- ample, in the 2014 Workshop on Statistical Ma- chine Translation (WMT), which since 2012 has provided a main venue for evaluation of systems, sentence-level systems were evaluated with re- spect to three distinct gold standard representa- tions and each of those compared to predictions using four different measures, resulting in a total of 12 different system rankings, 6 identified as of- ficial rankings (Bojar et al., 2014). Although the aim of several methods of evalua- tion is to provide more insight into performance of systems, this also produces conflicting results and raises the question which method of evaluation re- ally identifies the system(s) or method(s) that best predicts translation quality. For example, an ex- treme case in WMT-14 occurred for sentence-level quality estimation for English-to-Spanish. In each of the 12 system rankings, many systems were tied and this resulted in a total of 22 official winning systems for this language pair. Besides leaving po- tential users of quality estimation systems at a loss as to what the best system may be, a large number of inconclusive evaluation methodologies is also likely to lead to confusion about which evaluation methods should be …

Content truncated to 500 words