What’s the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT

What’s the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT Lukas Fischer 1 Samuel Läubli 1,2 1 TextShuttle AG 2 Department of Computational Linguistics, University of Zurich Abstract Machine translation (MT) has been shown to produce a number of errors that re- quire human post-editing, but the extent to which professional human translation (HT) contains such errors has not yet been compared to MT. We compile pre- translated documents in which MT and HT are interleaved, and ask professional trans- lators to flag errors and post-edit these doc- uments in a blind evaluation. We find that the post-editing effort for MT segments is only higher in two out of three language pairs, and that the number of segments with wrong terminology, omissions, and typographical problems is similar in HT. 1 Introduction Machine translation (MT) quality has improved substantially over the past years, allegedly to the degree that it is no longer distinguishable from pro- fessional human translation (HT). The first claims of human–machine parity were based on MT sys- tems geared to news translation (Hassan et al., 2018; Popel, 2018), and soon refuted due to weak- nesses in the evaluation methodology. Repro- ductions with professional translators rather than crowd workers and full documents rather than sin- gle sentences likewise concluded that HT was su- perior to MT in terms of both accuracy and fluency (Toral et al., 2018; Läubli et al., 2018). Human–machine parity claims may not hold with MT systems for broad domains such as news articles, but systems geared to narrower domains c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. have been shown to achieve far better quality (e.g., Levin et al., 2017), and it is unclear how they compare to specialised human professionals. In this paper, we propose an evaluation design that avoids the weaknesses identified in previous human–machine comparisons (Section 2), and re- lies on metrics that are arguably better quantifiable and interpretable than adequacy and fluency judg- ments: error counts and edit distance (Section 2.2). Evaluators are asked to flag errors in and post-edit full documents, where half of the sentences are MT and the other half are HT (Section 3). We analyse data collected in a study involving three language pairs and ten professional translators, and find that professional translators post-edit professional HT almost as much as MT, and rate the two simi- larly in terms of issues with terminology, omis- sion, and typography (Section 4). We also contex- tualise our results within the ongoing discussion on human–machine parity, suggesting that further as- sessments will need to focus specifically on what professional translators can do better than MT sys- tems – and vice versa – rather than comparing their “overall quality” (Section 5). Our method should provide a means to assess the viability of MT in specific professional translation contexts, and may possibly help decrease resistance against the tech- nology among professional translators. 2 Background How …

Content truncated to 500 words