Further Investigation into Reference Bias in Monolingual Evaluation of Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2476–2485 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Further Investigation into Reference Bias in Monolingual Evaluation of Machine Translation Qingsong Ma∗† Yvette Graham† Timothy Baldwin‡ Qun Liu† ∗Institute of Computing Technology Chinese Academy of Sciences maqingsong@ict.ac.cn †ADAPT Centre Dublin City University firstname.surname@dcu.ie ‡Computing and Info Systems University of Melbourne tb@ldwin.net Abstract Monolingual evaluation of Machine Translation (MT) aims to simplify human assessment by requiring assessors to com- pare the meaning of the MT output with a reference translation, opening up the task to a much larger pool of genuinely qual- ified evaluators. Monolingual evaluation runs the risk, however, of bias in favour of MT systems that happen to produce translations superficially similar to the ref- erence and, consistent with this intuition, previous investigations have concluded monolingual assessment to be strongly biased in this respect. On re-examination of past analyses, we identify a series of potential analytical errors that force some important questions to be raised about the reliability of past conclusions, however. We subsequently carry out further investi- gation into reference bias via direct human assessment of MT adequacy via quality controlled crowd-sourcing. Contrary to both intuition and past conclusions, results show no significant evidence of reference bias in monolingual evaluation of MT. 1 Introduction Despite it being known for some time now that au- tomatic metrics, such as BLEU (Papineni et al., 2002), provide a less than perfect substitute for human assessment (Callison-Burch et al., 2006), evaluation in MT more often than not still com- prises BLEU scores. Besides increased time and resources required by the alternative, human eval- uation of systems, human assessment of MT faces additional challenges, in particular the fact that human assessors of translation quality tend to be highly inconsistent. In recent Conference on Ma- chine Translation (WMT) shared tasks, for exam- ple, manual evaluators complete a relative ranking (RR) of the output of five alternate MT systems, where they must rank the quality of competing translations from best to worst. Within this set-up, when presented with the same pair of MT output translations, human assessors often disagree with one another’s preference, and even their own pre- vious judgment about which translation is better (Callison-Burch et al., 2007; Bojar et al., 2016). Low levels of inter-annotator agreement in human evaluation of MT not only cause problems with re- spect to the reliability of MT system evaluations, but unfortunately have an additional knock-on ef- fect with respect to the meta-evaluation of metrics, in providing an unstable gold standard. As such, provision of a fair and reliable human evaluation of MT remains a high priority for empirical evalu- ation. Direct assessment (DA) (Graham et al., 2013, 2014, 2016) is a relatively new human evaluation approach that overcomes previous challenges with respect to lack of reliability of human judges. DA collects assessments of translations separately in the form of both fluency and adequacy on a 0–100 rating scale, and, by combination of …

Content truncated to 500 words