Automated Evaluation Metric for Terminology Consistency in MT

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 450 - 457 December 7-8, 2022 ©2022 Association for Computational Linguistics Automated Evaluation Metric for Terminology Consistency in MT Kirill Semenov and Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostranské námestí 25, 118 00 Prague, Czech Republic kir.semenow@yandex.ru bojar@ufal.mff.cuni.cz Abstract The most widely used metrics for machine translation tackle sentence-level evaluation. However, at least for professional domains such as legal texts, it is crucial to measure the con- sistency of translation of terms throughout the whole text. This paper introduces an automated metric for term consistency evaluation in machine trans- lation (MT). To demonstrate the metric’s per- formance, we used the Czech-to-English trans- lated texts from the ELITR 2021 agreement corpus and the outputs of the MT systems that took part in WMT21 and WMT22 News Tasks. We show different modes of our evaluation al- gorithm and try to interpret the differences in the ranking of the translation systems based on standard sentence-level metrics and our ap- proach. We also demonstrate that the pro- posed metric scores significantly differ from the widespread automated metric scores, and correlate with human assessment. 1 Introduction Throughout the last decade, the quality of machine translation (MT) has improved significantly, and it is becoming a common phenomenon for various neural MT (NMT) systems to get better scores in manual direct assessment and other metrics than reference human translations (Akhbardeh et al., 2021; Bojar et al., 2018). However, such figures are obtained when the MT outputs are evaluated on the sentence level (i.e., each sentence is assessed sepa- rately, without context); in document-level evalua- tion, human translations typically remain the best, although exceptions exist (Popel et al., 2020). We can explain this situation by the fact that most of the current state-of-the-art NMT systems translate documents sentence by sentence, which thus can provoke inconsistencies in the translation of differ- ent linguistic elements – from anaphoric pronouns to named entities and terminology. We focus on the latter. While term inconsistencies can be tolerable for the general spheres of communication, they are unacceptable for several professional domains, es- pecially legal texts, where the coherent usage of terms is the ultimate characteristic. In the case of the term translation in the legal domain, the goal of the MT system can be split into several parts: 1. To translate one source term to only one tar- get term (we will call this property “consis- tency”); 2. To ensure that every source term is mapped to a distinct target term (we will call this property “unambiguity”); 3. To ensure that the target term is an adequate translation of the source term in general. In this paper, we present a novel metric that fo- cuses on the consistency and unambiguity of terms, whereas measuring the third parameter, adequacy, is delegated to the mainstream automated metrics such as BLEU (Papineni et al., 2002) and chrF (Popovi´c, 2015). Our proposed metric can be ap- plied automatically, …

Content truncated to 500 words