DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7795 - 7816 December 7-11, 2022 ©2022 Association for Computational Linguistics DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages Gabriele Sarti, Arianna Bisazza, Ana Guerberof-Arenas, Antonio Toral Center for Language and Cognition (CLCG), University of Groningen {g.sarti, a.bisazza, a.guerberof.arenas, a.toral.ruiz}@rug.nl Abstract We introduce DivEMT, the first publicly avail- able post-editing study of Neural Machine Translation (NMT) over a typologically di- verse set of target languages. Using a strictly controlled setup, 18 professional translators were instructed to translate or post-edit the same set of English documents into Ara- bic, Dutch, Italian, Turkish, Ukrainian, and Vietnamese. During the process, their ed- its, keystrokes, editing times and pauses were recorded, enabling an in-depth, cross-lingual evaluation of NMT quality and post-editing ef- fectiveness. Using this new dataset, we assess the impact of two state-of-the-art NMT sys- tems, Google Translate and the multilingual mBART-50 model, on translation productivity. We find that post-editing is consistently faster than translation from scratch. However, the magnitude of productivity gains varies widely across systems and languages, highlighting major disparities in post-editing effectiveness for languages at different degrees of typologi- cal relatedness to English, even when control- ling for system architecture and training data size. We publicly release the complete dataset1 including all collected behavioral data, to fos- ter new research on the translation capabilities of NMT systems for typologically diverse lan- guages. 1 Introduction Recent advances in neural language modeling and multilingual training have prompted a widespread adoption of machine translation (MT) technologies across an unprecedented range of world languages. While the benefits of state-of-the-art MT for cross- lingual information access are undisputed (Lom- mel and Pielmeier, 2021), its usefulness as an aid to professional translators varies considerably across domains, subjects and language combina- tions (Zouhar et al., 2021). In the last decade, the 1https://github.com/gsarti/divemt https://huggingface.co/datasets/GroNLP/divemt MT community has been including an increasing number of languages in its automatic and human evaluation efforts (Bojar et al., 2013; Barrault et al., 2021). However, the results of these evaluations are typically not directly comparable across different language pairs for various reasons. First, reference- based automatic quality metrics are hardly compa- rable across different target languages (Bugliarello et al., 2020). Secondly, human judgments are col- lected independently for different language pairs, making their cross-lingual comparison vulnerable to confounding factors such as tested domains and training data sizes. Similarly, recent work on NMT post-editing efficiency has focused on specific lan- guage pairs such as English-Czech (Zouhar et al., 2021), German-Italian, German-French (Läubli et al., 2019) and English-Hindi (Ahsan et al., 2021), but a controlled comparison across a set of typolog- ically diverse languages is still lacking. In this work, we assess the usefulness of state- of-the-art NMT in professional translation with a strictly controlled cross-language setup (Figure 1). Specifically, professionals were asked to translate the same English documents into six typologically different languages (Arabic, Dutch, Italian, Turk- ish, Ukrainian, and Vietnamese) using the same platform and guidelines. Three translation …

Content truncated to 500 words