On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6018–6023 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6018 On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation Thierry Poibeau Laboratoire LATTICE CNRS & ´Ecole normale sup´erieure/PSL & Universit´e Sorbonne nouvelle 1, rue Maurice Arnoux – 92120 Montrouge, France thierry.poibeau@ens.psl.eu Abstract In this paper, we reassess claims of human parity and super human performance in machine translation. These terms have already been discussed, as well as the evaluation protocols used to achieved these conclusions (human-parity is achieved i) only for a very reduced number of languages, ii) on very specific types of documents and iii) with very literal translations). However, we think it is necessary to consider these questions again. We show that the terms used are themselves problematic, and that human translation involves much more than what is embedded in automatic systems. We also discuss ethical issues related to the way results are presented and advertised. Finally, we claim that a better assessment of human capacities should be put forward and that the goal of replacing humans by machines is not a desirable one. Keywords: Neural machine translation, evaluation, human parity, human factors, ethics, 1. Introduction Machine translation (MT) has recently achieved im- pressive results. Thanks to the success of deep learning approaches, state of the art systems have been made available online, which translate quite accurately be- tween a large variety of languages (Bojar et al., 2018; Barrault et al., 2019), although differences from one language to the other remain important. This success is clear, it has been widely acclaimed and end-users can feel this progress very directly in their everyday life. Whereas previous systems were providing translations that needed to be heavily revised and corrected, recent systems provide decent and readable translations most of the time. Of course, the quality depends on a lot of parameters, among others the language pair consid- ered, the type of text and the required quality of the final result. Following this success, several teams have claimed to have reached super human performance for different pairs of languages. The term “human parity” (Has- san et al., 2018) and even “super-human performance” (Barrault et al., 2019) have been used for example in the context of the Workshop on Machine Translation (WMT) evaluation campaigns1. This means that MT is supposed to be equal or even to have surpassed the quality of human translation, at least in this context. Practically, during the evaluation, the translation pro- duced by a machine was regularly preferred to the one obtained from a professional translator. This sounds impressive, but should be discussed further and put into perspective. In this paper, we claim that 1See for example: “English to German: Facebook-FAIR achieves super-human translation performance; several sys- tems are tied with human performance.” (Barrault et al., 2019) the term ”human parity” and, moreover, ”super human performance” are misleading, in that they concern only very specific types of texts, for …

Content truncated to 500 words