Quality Analysis of Multilingual Neural Machine Translation Systems and Reference Test Translations for the English-Romanian language pair in the Medical Domain

Quality Analysis of Multilingual Neural Machine Translation Systems and Reference Test Translations for the English-Romanian language pair in the Medical Domain Miguel Rios, Alina Secar˘a, Raluca-Maria Chereji, Dragos, Ciobanu Centre for Translation Studies, University of Vienna {miguel.angel.rios.gaona, alina.secara, raluca-maria.chereji, dragos.ioan.ciobanu}@univie.ac.at Abstract Multilingual Neural Machine Translation (MNMT) models allow translation across multiple languages based on a single sys- tem. We study the quality of a domain- adapted MNMT model in the medical do- main for English-Romanian with automatic metrics and a human error typology anno- tation based on the Multidimensional Qual- ity Metrics (MQM) framework. We fur- ther expand the MQM typology to include terminology-specific error categories. We compare the out-of-domain MNMT with the in-domain adapted MNMT on a stan- dard test dataset of abstracts from medical publications. The in-domain MNMT model outperforms the out-of-domain MNMT in all measured automatic metrics, and pro- duces fewer errors. We also manually anno- tate the reference test dataset to study the quality of the reference translations, and we identify a high number of omissions, ad- ditions, and mistranslations. We therefore question the assumed accuracy of existing datasets. Finally, we compare the corre- lation between the COMET, BERTScore, and chrF automatic metrics with the MQM annotated translations; COMET shows a better correlation with the MQM scores. 1 Introduction Neural Machine Translation (NMT) models have achieved competitive performance on low-resource language pairs, particularly for non-specialised do- mains (Araabi and Monz, 2020). However, in a © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. high-risk and low-resource domain, like the medi- cal domain, the accurate translation of terminology, alongside the absence of hallucinations and mis- translations are crucial for exchanging information across international healthcare providers or users (Skianis et al., 2020). Multilingual NMT (MNMT) models leverage many language pairs and millions of segments (Johnson et al., 2017) within one sys- tem. The inclusion of many language pairs helps to improve the translation quality for low-resource languages by transferring knowledge from high- resource languages via similar cross-lingual word representations. Moreover, domain adaptation tech- niques are used to adapt MNMT models into new domains (Bérard et al., 2020). However, evaluation studies of MNMT models are focused on automatic metrics without providing insights into the quality of the translation of a specialised domain. These automatic metrics require high-quality reference translations which reflect the specialised terminol- ogy and style of a given domain, but such transla- tions are difficult to find. Also, given that transla- tion processes and expertise vary among translators and other text producers, the quality of datasets in different language pairs can differ considerably. In addition, justifiable differences between source and target sentence content are caused by legitimate pragmatic translation strategies. Overall, automatic or even semi-automatic translation data gathering processes are not sophisticated enough yet to im- prove the quality of the source and/or target content, or filter out content mismatches between source and target sentences before aligning them. In this paper, we study the quality of …

Content truncated to 500 words