NMT or SMT: Case Study of a Narrow-domain English-Latvian Post-editing Project

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 373–383, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP NMT or SMT: Case Study of a Narrow-domain English-Latvian Post-editing Project Inguna Skadin¸a and M¯arcis Pinnis Tilde, Vienibas gatve 75A, Riga, Latvia {inguna.skadina,marcis.pinnis}@tilde.lv Abstract The recent technological shift in machine translation from statistical machine trans- lation (SMT) to neural machine transla- tion (NMT) raises the question of the strengths and weaknesses of NMT. In this paper, we present an analysis of NMT and SMT systems’ outputs from narrow domain English-Latvian MT systems that were trained on a rather small amount of data. We analyze post-edits produced by professional translators and manually an- notated errors in these outputs. Analysis of post-edits allowed us to conclude that both approaches are comparably success- ful, allowing for an increase in transla- tors’ productivity, with the NMT system showing slightly worse results. Through the analysis of annotated errors, we found that NMT translations are more fluent than SMT translations. However, errors re- lated to accuracy, especially, mistransla- tion and omission errors, occur more often in NMT outputs. The word form errors, that characterize the morphological rich- ness of Latvian, are frequent for both sys- tems, but slightly fewer in NMT outputs. 1 Introduction For many years, the central problem in machine translation (MT) has been the quality. MT qual- ity has been recognized as a complicated research question when translation is performed into a mor- phologically rich (and also under-resourced) lan- guage with a relatively free word order, e.g., Bul- garian, Croatian, Estonian, Finnish, Greek or Lat- vian. Possible solutions for widely used statistical machine translation have been studied for many years (e.g., Koehn and Hoang 2007; Tamchyna and Bojar 2013; Burlot and Yvon 2015). Today machine translation is experiencing a paradigm shift from (phrase-based) statistical ma- chine translation (SMT) to neural machine trans- lation (NMT). The first results obtained in recent years are promising, as it can be seen from the results of WMT 2016 (Bojar et al., 2016) and WMT 2017 (Bojar et al., 2017). As NMT becomes more and more popular, the question of what can we expect from NMT in terms of quality becomes very important. Recent analysis of English to German SMT and NMT outputs of manual transcripts of short speeches showed that NMT can decrease the post-editing effort (Bentivogli et al., 2016). A comparison of NMT and SMT systems for nine language direc- tions (English to and from Czech, German, Ro- manian, Russian, and English to Finnish) on news stories made by Toral and S´anchez-Cartagena (2017) showed that translations produced by NMT systems are more fluent and more accurate in terms of word order compared to translations pro- duced by SMT systems. By analyzing of man- ually error-annotated outputs of generic English- Croatian MT systems, Klubiˇcka et al. (2017) found that NMT handles all types of agreement better than SMT (including factored models). In this paper, we delve further into analyzing the strengths and weaknesses …

Content truncated to 500 words