A fine-grained error analysis of NMT, SMT and RBMT output for English-to-Dutch

A Fine-grained Error Analysis of NMT, PBMT and RBMT Output for English- to-Dutch Laura Van Brussel, Arda Tezcan, Lieve Macken Ghent University, Department of Translation, Interpreting and Communication Groot-Brittanniëlaan 45, 9000 Ghent, Belgium laura.vanbrussel@ugent.be, arda.tezcan@ugent.be, lieve.macken@ugent.be Abstract This paper presents a fine-grained error comparison of the English-to-Dutch translations of a commercial neural, phrase-based and rule- based machine translation (MT) system. For phrase-based and rule-based machine translation, we make use of the annotated SCATE corpus of MT errors, enriching it with the annotation of neural MT errors and updating the SCATE error taxonomy to fit the neural MT output as well. Neural, in general, outperforms phrase-based and rule-based systems especially for fluency, except for lexical issues. On the accuracy level, the improvements are less obvious. The target sentence does not always contain traces or clues of content being missing (omissions). This has repercussions for quality estimation or gisting operating only on the monolingual level. Mistranslations are part of another well represented error category, comprising a high number of word-sense disambiguation errors and a variety of other mistranslation errors, making it more complex to annotate or post-edit. Keywords: machine translation, error classification, bilingual corpus 1. Introduction Since 2016, the landscape of automated translation has substantially changed with the arrival of neural machine translation (NMT). The output quality of this newest system is a hot topic for research at the moment. It has already been compared with the previous state-of-the-art phrase-based machine translation (PBMT) engines and even with rule-based machine translation (RBMT) engines, focusing on the overall performance by applying various automatic metrics, by manual ranking and scoring (Shterionov, Casanellas, Superbo, & O’Dowd, 2017), post- editing or manual error classification (Bentivogli, Bisazza, Cettolo, & Federico, 2016). The scope of the studies range from one to multiple language directions (Toral and Sánchez-Cartagena 2017; Klubička, Toral, and Sánchez- Cartagena 2017; Bojar et al. 2016). Unlike previous work, where engines are developed in research institutes or test suites are built for evaluation, in this paper, we take a different angle by using commercial MT systems and real- life texts from different genres, and thus bring more ecological validity into the field. In this article, we compare the output of commercial NMT, PBMT and RBMT systems for English to Dutch. Since it provides a detailed overview of the types of errors, we want to discover if the findings for other language pairs apply to English-to-Dutch as well, identify the actual improvements that NMT systems bring to automated translation and get a grip on their potential shortcomings. 2. Related Work This analysis is carried out in the framework of the SCATE project (Tezcan, Hoste, & Macken, 2017b) and draws on its corpus of PBMT and RBMT errors. We used SCATE’s error taxonomy to annotate the same sentences, this time translated by Google’s Neural Machine Translation (GNMT)1. A substantial part of the research in the field focuses on the language pair English-German. For English to German, Bentivogli et al. (2016) found that NMT output contains less lexical, morphological and …

Content truncated to 500 words