Findings of the WMT Shared Task on Machine Translation Using Terminologies

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 652–663 November 10–11, 2021. ©2021 Association for Computational Linguistics 652 Findings of the WMT Shared Task on Machine Translation Using Terminologies Md Mahfuz ibn Alam1, Ivana Kvapilíková5, Antonios Anastasopoulos1, Laurent Besacier2, Georgiana Dinu3, Marcello Federico3, Matthias Gallé2, Philipp Koehn4, Vassilina Nikoulina2, Kweon Woo Jung2 1Department of Computer Science, George Mason University 2NAVER 3AWS 4Facebook / Johns Hopkins University 5Charles University {malam21, antonis}@gmu.edu Abstract Language domains that require very careful use of terminology are abundant and reflect a significant part of the translation industry. In this work we introduce a benchmark for eval- uating the quality and consistency of terminol- ogy translation, focusing on the medical (and COVID-19 specifically) domain for five lan- guage pairs: English to French, Chinese, Rus- sian, and Korean, as well as Czech to Ger- man. We report the descriptions and results of the participating systems, commenting on the need for further research efforts towards both more adequate handling of terminologies as well as towards a proper formulation and evaluation of the task. 1 Introduction Language domains that require very careful use of terminology are abundant. The need to adequately translate within such domains is undeniable, as shown by e.g. the different WMT shared tasks on biomedical translation. More interestingly, as the abundance of research on domain adaptation shows, such language do- mains are (a) not adequately covered by existing data and models, while (b) new (or “surge") do- mains arise and models need to be adapted, often with significant downstream implications: consider the new COVID-19 domain and the large efforts for translation of critical information regarding pan- demic handling and infection prevention strategies. In the case of newly developed domains, while parallel data are hard to come by, it is fairly straight- forward to create word- or phrase-level terminolo- gies, which can be used to guide professional trans- lators and ensure both accuracy and consistency. This shared task1 replicated such a scenario, and invited participants to explore methods to incorpo- rate terminologies into either the training or the 1http://statmt.org/wmt21/terminology-task.html inference process, in order to improve both the ac- curacy and consistency of MT systems on a new domain. 2 Shared Task Details The shared task focused on five language pairs, with systems evaluated on: • English to French • English to Chinese • English to Russian • English to Korean • Czech to German The last three language pairs were “surprise" lan- guage pairs. This shared task construction follows a three-phase approach to ensure the generalizabil- ity of the findings, inspired by other multilingual shared tasks (Vylomova et al., 2020). In this set- ting, only part of the evaluation language pairs (or languages) are revealed from the beginning (the Development Phase). In this elongate period (a couple of months), the participants are provided with data in some language pairs to develop their methods. The second phase is the Generalization phase, which is a short time period (two to three weeks in this task’s case), in which …

Content truncated to 500 words