Is all that Glitters in Machine Translation Quality Estimation really Gold?

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3124–3134, Osaka, Japan, December 11-17 2016. Is all that Glitters in Machine Translation Quality Estimation really Gold? Yvette Graham∗ Timothy Baldwin† Meghan Dowling∗ Maria Eskevich‡ Teresa Lynn∗ Lamia Tounsi∗ ∗ADAPT Centre Dublin City University firstname.surname@dcu.ie †Computing and Info Systems University of Melbourne tb@ldwin.net ‡Centre for Language Studies Radboud University m.eskevich@let.ru.nl Abstract Human-targeted metrics provide a compromise between human evaluation of machine transla- tion, where high inter-annotator agreement is difficult to achieve, and fully automatic metrics, such as BLEU or TER, that lack the validity of human assessment. Human-targeted translation edit rate (HTER) is by far the most widely employed human-targeted metric in machine transla- tion, commonly employed, for example, as a gold standard in evaluation of quality estimation. Original experiments justifying the design of HTER, as opposed to other possible formulations, were limited to a small sample of translations and a single language pair, however, and this mo- tivates our re-evaluation of a range of human-targeted metrics on a substantially larger scale. Results show significantly stronger correlation with human judgment for HBLEU over HTER for two of the nine language pairs we include and no significant difference between correlations achieved by HTER and HBLEU for the remaining language pairs. Finally, we evaluate a range of quality estimation systems employing HTER and direct assessment (DA) of translation adequacy as gold labels, resulting in a divergence in system rankings, and propose employment of DA for future quality estimation evaluations. 1 Introduction Although human evaluation of translation quality in theory provides the most meaningful assessment of machine translation (MT), achieving high levels of agreement between human assessors has proven chal- lenging. For example, the annual Workshop on Statistical Machine Translation (WMT) provides large- scale human evaluation of systems, and reports inter-annotator agreement ranging from 0.260 (WMT- 13) to 0.405 (WMT-15), with intra-annotator agreement not faring much better, from 0.407 (WMT-12) to 0.595 (WMT-15) (Callison-Burch et al., 2012; Bojar et al., 2013; Bojar et al., 2015).1 Low agree- ment levels in human annotation cause challenges for tasks that require evaluation of MT output on the segment-level. For example, evaluation of MT quality estimation (QE) requires the comparison of sys- tem predictions of translation quality with segment-level human assessment. Automatic metrics, such as BLEU or TER, although fully repeatable, unfortunately lack the validity of human assessment and are well-known to suffer from bias in favour of translations that happen to be superficially similar to reference translations. Part-automatic human-targeted metrics, however, are commonly employed as a substitute for hu- man assessment on the segment-level, as they appear to provide a happy medium between fully auto- matic metrics, that lack the validity of human assessment, and human assessment that lacks reliabil- ity/reproducibility of automatic metrics. Human-targeted reference translations, each manually created by minimally post-editing the individual MT output translation to be assessed, remove the bias usually introduced by comparison with a generic reference. Subsequently an automatic component is applied to quantify the error …

Content truncated to 500 words