Assessing Human-Parity in Machine Translation on the Segment Level

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4199–4207 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4199 Assessing Human-Parity in Machine Translation on the Segment Level Yvette Graham ADAPT Trinity College Dublin ygraham@tcd.ie Christian Federmann Microsoft Research chrife@microsoft.com Maria Eskevich CLARIN ERIC Utrecht maria@clarin.eu Barry Haddow School of Informatics University of Edinburgh bhaddow@inf.ed.ac.uk Abstract Recent machine translation shared tasks have shown top-performing systems to tie or in some cases even outperform human transla- tion. Such conclusions about system and hu- man performance are, however, based on es- timates aggregated from scores collected over large test sets of translations and so leave some remaining questions unanswered. For in- stance, simply because a system significantly outperforms the human translator on average may not necessarily mean that it has done so for every translation in the test set. Further- more, are there remaining source segments present in evaluation test sets that cause sig- nificant challenges for top-performing systems and can such challenging segments go unno- ticed due to the opacity of current human eval- uation procedures? To provide insight into these issues we carefully inspect the outputs of top-performing systems in the recent WMT19 news translation shared task for all language pairs in which a system either tied or outper- formed human translation. Our analysis pro- vides a new method of identifying the remain- ing segments for which either machine or hu- man perform poorly. For example, in our close inspection of WMT19 English to German and German to English we discover the segments that disjointly proved a challenge for human and machine. For English to Russian, there were no segments included in our sample of translations that caused a significant challenge for the human translator, while we again iden- tify the set of segments that caused issues for the top-performing system. 1 Introduction Recent results of machine translation evaluation shared tasks indicate that state-of-the-art is now achieving and possibly even surpassing human per- formance, with the most recent annual Conference on Machine translation (WMT) news task provid- ing extensive human evaluation of systems, con- cluding that several systems performed on average as well as human for English to Russian, English to German and German to English translation and a top system even surpassed human performance for the last two language pairs. Since 2017 the official results of the WMT news tasks have been based on the human evaluation methodology known as Direct Assessment (DA) (Graham et al., 2016), due to its many advantages over older technologies. DA, for example, includes quality control mechanisms that allow data col- lected anonymously from crowd-sourced workers to be filtered according to reliability.1 Although WMT news task results are admittedly based on substantially more valid methodology than those usually found in general in system comparisons using automatic metrics such as BLEU, results in WMT human evaluations still leave some ques- tions unanswered. For example, DA scores are based on average ratings attributed to translations sampled from large test sets, and although such …

Content truncated to 500 words