Quantitative Analysis of Post-Editing Effort Indicators for NMT

Quantitative Analysis of Post-Editing Effort Indicators for NMT Sergi Alvarez Universitat Pompeu Fabra salvarezvid@uoc.edu Antoni Oliver Universitat Oberta de Catalunya aoliverg@uoc.edu Toni Badia Universitat Pompeu Fabra toni.badia@upf.edu Abstract The recent improvements in machine translation (MT) have boosted the use of post-editing (PE) in the translation indus- try. A new MT paradigm, neural MT (NMT), is displacing its corpus-based pre- decessor, statistical machine translation (SMT), in the translation workflows cur- rently implemented because it usually in- creases the fluency and accuracy of the MT output. However, usual automatic mea- surements do not always indicate the qual- ity of the MT output and there is still no clear correlation between PE effort and productivity. We present a quantitative analysis of different PE effort indicators for two NMT systems (transformer and seq2seq) for English-Spanish in-domain medical documents. We compare both sys- tems and study the correlation between PE time and other scores. Results show less PE effort for the transformer NMT model and a high correlation between PE time and keystrokes. 1 Introduction The use of machine translation (MT) systems for the production of drafts that are later post-edited has become a widespread practice in the transla- tion industry. Research has concluded that post- editing of machine translation (PEMT) is usually more efficient than translating from scratch (Plitt and Masselot, 2010; Federico et al., 2012; Green et al., 2013). Thus, it has been included in the trans- lation workflow because it increases productivity c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. when compared with human translation (Aranberri et al., 2014) and reduces costs (Guerberof, 2009) without having a negative impact on quality (Plitt and Masselot, 2010). Post-editors “edit, modify and/or correct pre-translated text that has been pro- cessed by an MT system from a source language into (a) target language(s)” (Allen, 2003, p. 296). In recent years, neural machine translation (NMT) has produced promising results in terms of quality, for example in WMT 2019 (Barrault et al., 2019). This has increased the interest in this new paradigm for the translation industry, which has begun to substitute its corpus-based predecessor, statistical machine translation (SMT), with new NMT models. It has also boosted the incorpo- ration of PEMT in many translation workflows. In the 2018 Language Industry Survey,1 37% of the respondents reported an increase of MT post- editing and an additional 17% indicated that they had started implementing this practice. Given the improved-quality performance of NMT and its widespread use in industrial scenar- ios, it is necessary to study the potential this ap- proach can offer to post-editing. One of the main problems is that automatic scores give a general idea of the MT output quality but do not always correlate to post-editing effort (Koponen, 2016; Shterionov et al., 2018). However, many profes- sional translators state that if the quality of the MT output is not good enough, they delete the remain- ing segments and translate everything from scratch (Parra Escart´ın and …

Content truncated to 500 words