Literary Machine Translation under the Magnifying Glass: Assessing the Quality of an NMT-Translated Detective Novel on Document Level

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3790–3798 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3790 Literary Machine Translation under the Magnifying Glass: Assessing the Quality of an NMT-Translated Detective Novel on Document Level Margot Fonteyne, Arda Tezcan, Lieve Macken LT3, Language and Translation Technology Team, Ghent University Groot-Brittanni¨elaan 45, 9000 Ghent, Belgium {mafontey.fonteyne, arda.tezcan, lieve.macken}@ugent.be Abstract Several studies (covering many language pairs and translation tasks) have demonstrated that translation quality has improved enor- mously since the emergence of neural machine translation systems. This raises the question whether such systems are able to produce high-quality translations for more creative text types such as literature and whether they are able to generate coherent translations on document level. Our study aimed to investigate these two questions by carrying out a document-level evaluation of the raw NMT output of an entire novel. We translated Agatha Christie’s novel The Mysterious Affair at Styles with Google’s NMT system from English into Dutch and annotated it in two steps: first all fluency errors, then all accuracy errors. We report on the overall quality, determine the remaining issues, compare the most frequent error types to those in general-domain MT, and investigate whether any accuracy and fluency errors co-occur regularly. Additionally, we assess the inter-annotator agreement on the first chapter of the novel. Keywords: literary machine translation, quality assessment, document-level evaluation 1. Introduction It is widely accepted that with the advent of neural ma- chine translation (NMT) translation quality has made a big leap forward. Several studies, covering many language pairs and translation tasks, using both human and auto- matic evaluation methods, have demonstrated that NMT systems outperform (the previous state-of-the-art) statisti- cal MT (Bentivogli et al., 2016; Burchardt et al., 2017; Toral and S´anchez-Cartagena, 2017; Klubiˇcka et al., 2018; Van Brussel et al., 2018; Shterionov et al., 2018; Jia et al., 2019; Daems and Macken, 2019). Hassan et al. (2018) even claimed that, based on sentence-level evaluation pro- tocols, Microsoft’s NMT systems achieved human parity on the translation of Chinese to English news texts. L¨aubli et al. (2018), however, carried out document-level eval- uations on the same data set and found that professional translators preferred human over machine translations, em- phasizing the need to shift to document-level evaluations as machine translation quality improves. NMT’s quality im- provements have also aroused interest in the use of MT for more creative text types such as literature (Toral and Way, 2018; Kuzman et al., 2019; Matusov, 2019). This study aims to assess whether a general-domain NMT system is able to produce high-quality translations for literary trans- lation taking into account document-level aspects of trans- lation such as coherence and cohesion. We evaluate the MT output of Agatha Christie’s novel The Mysterious Affair at Styles, which was translated by Google’s NMT system (GNMT) from English into Dutch. For the case study by Tezcan et al. (2019), which focuses on how the MT differs from the published human transla- tion …

Content truncated to 500 words