Neural Machine Translation Quality and Post-Editing Performance

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10204–10214 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10204 Neural Machine Translation Quality and Post-Editing Performance Vil´em Zouhar,⋆ Aleˇs Tamchyna,† Martin Popel,⋆ Ondˇrej Bojar⋆ Charles University, Faculty of Mathematics and Physics⋆ Institute of Formal and Applied Linguistics Malostransk´e N´amˇest´ı 25, 118 00 Prague, Czech Republic {zouhar, popel, bojar}@ufal.mff.cuni.cz Memsource† Sp´alen´a 108/51, Prague, Czech Republic ales.tamchyna@memsource.com Abstract We test the natural expectation that using MT in professional translation saves human pro- cessing time. The last such study was car- ried out by Sanchez-Torron and Koehn (2016) with phrase-based MT, artificially reducing the translation quality. In contrast, we focus on neural MT (NMT) of high quality, which has become the state-of-the-art approach since then and also got adopted by most translation companies. Through an experimental study involv- ing over 30 professional translators for English→Czech translation, we examine the relationship between NMT performance and post-editing time and quality. Across all mod- els, we found that better MT systems indeed lead to fewer changes in the sentences in this industry setting. The relation between system quality and post-editing time is however not straightforward and, contrary to the results on phrase-based MT, BLEU is definitely not a stable predictor of the time or final output quality. 1 Introduction Machine translation is increasingly utilized in the translation and localization industry. One of the most common use cases is MT post-editing where human translators use MT outputs as a starting point and make edits to obtain the final translation. This process has been shown to be more efficient than translating from scratch, i.e. it and can lead to reductions in cost and delivery time (Plitt and Masselot, 2010). While various studies have looked at how MT post-editing affects translation quality and speed in general, few have attempted to measure how the quality of MT outputs affects the productivity of translators. In this work, we attempt to answer the following questions: • How strong is the relationship between MT quality and post-editing speed? • Does MT quality have a measurable impact on the quality of the post-edited translation? • Is the effect of MT quality still persistent in a second round of post-editing (“revision”)? • Is the post-editing process different when hu- man translation is used instead of MT as the input? • How large are the edits in the different rounds of post-editing? We have carried out a large-scale study on one language pair that involved over 30 professional translators and translation reviewers, who worked in two stages to post-edit outputs of 13 different sources (11 MT engines, 1 raw source, 1 human reference). This allowed us to collect not only the post-editing times but also to estimate the quality of the produced results. Based on this data, we present an in-depth analysis and provide observa- tions and recommendations for utilizing MT in lo- calization workflows. While the task for humans for both rounds is the same (improve a given trans- lation by …

Content truncated to 500 words