Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs

Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs Maria Stasimioti Vilelmini Sosoni Department of Foreign Languages, Translation and Interpreting Ionian University Corfu, Greece stasimioti@ionio.gr sosoni@ionio.gr Despoina Mouratidis Katia Kermanidis Department of Informatics Ionian University Corfu, Greece c12mour@ionio.gr kerman@ionio.gr Abstract The present study aims to compare three systems: a generic statistical machine translation, a generic neural machine translation and a tailored-NMT system focusing on the English to Greek language pair. The comparison is carried out following a mixed-methods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post editing effort and human error analysis performed by 16 postgraduate Translation students. The findings reveal a higher score for both the generic NMT and the tailored-NMT outputs as regards automatic metrics and human evaluation metrics, with the tailored-NMT output faring even better than the generic NMT output. 1 Introduction Latest technological advances in machine translation (MT) have led to a wider availability of MT systems for various language pairs and neural machine translation (NMT) has been widely hailed as a significant development in the improvement of the quality of MT, given that NMT models have been proven to consistently outperform statistical machine translation (SMT) models in shared tasks, as well as in various project outcomes (Toral and Sánchez-Cartagena, 2017; Castilho et al., 2017a, 2017b, 2018; Klubička et al., 2017, 2018; Popović, 2017, 2018). MT has been moved “from the peripheries of the translation field closer to the centre” (Koponen, 2016a, p. 131) and has been integrated in the translation workflow, by using machine translated text as a raw translation to be further post-edited by a translator (Lommel and DePalma, 2016; Koponen, 2016b). The differences between various MT systems, as regards the quality of their output and the types of errors included therein, have been reported by several recent studies. Some (Bahdanau et al., 2015; Jean et al., 2015; Junczys-Dowmunt, 2016; Dowling et al., 2018) relied on automatic evaluation metrics (AEMs) like BLEU (Papineni et al., 2002) and HTER (Snover et al., 2006); others used human evaluations of the MT output quality, employing adequacy and fluency ratings (Bentivogli et al., 2016), manual error analyses (Klubička et al., 2017, 2018; Popović, 2018) or a combination of methods (Burchardt et al., 2017; Castilho et al., 2017a, 2017b, 2018; Toral and Sánchez-Cartagena, 2017; Shterionov et al., 2018; Koponen et al., 2019; Jia et al., 2019; Stasimioti and Sosoni, 2019). Drawing on these studies, the present study aims to compare three systems: a generic SMT, a generic NMT and a tailored-NMT system, namely a factored or custom-trained NMT system, focusing on the English to Greek language pair. The comparison is performed following a mixed- methods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post-editing (PE) effort and human error analysis. Tο the best of our knowledge there are no studies to date for the English to Greek language pair which compare generic and custom-trained MT systems, while there are …

Content truncated to 500 words