Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 113–123 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64012 Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation Antonio Toral Center for Language and Cognition University of Groningen The Netherlands a.toral.ruiz@rug.nl Sheila Castilho Ke Hu Andy Way ADAPT Centre Dublin City University Ireland firstname.secondname@adaptcentre.ie Abstract We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the transla- tion of news from Chinese into English, using pairwise ranking and considering three vari- ables that were not taken into account in that previous study: the language in which the source side of the test set was originally writ- ten, the translation proficiency of the evalua- tors, and the provision of inter-sentential con- text. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that human parity has not been achieved. We compare the judgments of professional trans- lators against those of non-experts and dis- cover that those of the experts result in higher inter-annotator agreement and better discrim- ination between human and machine transla- tions. In addition, we analyse the human trans- lations of the test set and identify important translation issues. Finally, based on these find- ings, we provide a set of recommendations for future human evaluations of MT. 1 Introduction Neural machine translation (NMT) has revolu- tionised the field of MT by overcoming many of the weaknesses of the previous state-of-the-art phrase-based machine translation (PBSMT) (Ben- tivogli et al., 2016; Toral and S´anchez-Cartagena, 2017). In only a few years since the first work- ing models, this approach has led to a substan- tial improvement in translation quality, reported in terms of automatic metrics (Bojar et al., 2016, 2017; Sennrich et al., 2016). This has ignited higher levels of expectation, fuelled in part by hy- perbolic claims from large MT developers. First we saw in Wu et al. (2016) that Google NMT was “bridging the gap between human and ma- chine translation [quality]”. This was amplified recently by the claim by Hassan et al. (2018) that Microsoft had ”achieved human parity” in terms of translation quality on news translation from Chinese to English, and more recently still by SDL who claimed to have “cracked” Russian-to- English NMT with “near perfect” translation qual- ity.1 However, when human evaluation is used to compare NMT and SMT, the results do not always favour NMT (Castilho et al., 2017a,b). Accompanying the claims regarding the capa- bility of the Microsoft Chinese-to-English NMT system, Hassan et al. (2018) released their exper- imental data2 which permits replicability of their experiments. In this paper, we provide a detailed examination of Microsoft’s claim to have reached human parity for the task of translating news from Chinese (ZH) to English (EN). They provide two definitions in this regard, namely: Definition 1. If a bilingual human judges the quality …

Content truncated to 500 words