Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019

Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019 Antonio Toral Center for Language and Cognition University of Groningen The Netherlands a.toral.ruiz@rug.nl Abstract We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English→German, English→Russian and German→English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersentential context available, (ii) the limited transla- tion proficiency of the evaluators and (iii) the use of a reference translation. We then conduct a modified evaluation taking these issues into account. Our results indicate that all the claims of human parity and super-human performance made at WMT 2019 should be refuted, except the claim of human parity for English→German. Based on our findings, we put forward a set of recommendations and open questions for future assessments of human parity in machine translation. 1 Introduction The quality of the translations produced by ma- chine translation (MT) systems has improved con- siderably since the adoption of architectures based on neural networks (Bentivogli et al., 2016). To the extent that, in the last two years, there have been claims of MT systems reaching human parity and even super-human performance (Hassan et al., 2018; Bojar et al., 2018; Barrault et al., 2019). Fol- lowing Hassan et al. (2018), we consider that hu- man parity is achieved for a given task t if the per- formance attained by a computer on t is equivalent c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. to that of a human, i.e. there is no significant dif- ference between the performance obtained by hu- man and by machine. Super-human performance is achieved for t if the performance achieved by a computer is significantly better than that of a hu- man. Two claims of human parity in MT were re- ported in 2018. One by Microsoft, on news trans- lation for Chinese→English (Hassan et al., 2018), and another at the news translation task of WMT for English→Czech (Bojar et al., 2018), in which MT systems Uedin (Haddow et al., 2018) and Cuni-Transformer (Kocmi et al., 2018) reached human parity and super-human performance, re- spectively. In 2019 there were additional claims at the news translation task of WMT (Barrault et al., 2019): human parity for German→English, by several of the submitted systems, and for English→Russian, by system Facebook-FAIR (Ng et al., 2019), as well as super-human performance for English→German, again by Facebook-FAIR. The claims of human parity and super-human performance in MT made in 2018 (Hassan et al., 2018; Bojar et al., 2018) have been since refuted given three issues in their evaluation setups (L¨aubli et al., 2018; Toral et al., 2018): (i) part of the source text of the test set was not original text but translationese, (ii) the sentences were evalu- ated in isolation, and (iii) the evaluation was not conducted by translators. …

Content truncated to 500 words