Coming to Terms with Glossary Enforcement: A Study of Three Approaches to Enforcing Terminology in NMT

Coming to Terms with Glossary Enforcement: A Study of Three Approaches to Enforcing Terminology in NMT Fred Bane Anna Zaretskaya Tània Blanch Miró Celia Soler Uguet João Torres TransPerfect {fbane,azaretskaya,tblanch,csuguet,joao.torres}@translations.com Abstract Enforcing terminology constraints is less straight-forward in neural machine trans- lation (NMT) than statistical machine translation. Current methods, such as alignment-based insertion or the use of factors or special tokens, each have their strengths and drawbacks. We describe the current state of research on terminology en- forcement in transformer-based NMT mod- els, and present the results of our investiga- tion into the performance of three different approaches. In addition to reference based quality metrics, we also evaluate the lin- guistic quality of the translations thus pro- duced. Our results show that each approach is effective, though a negative impact on translation fluency remains evident. 1 Introduction Ensuring translations use the preferred term can be business-critical for commercial translation providers. While there are existing methods to ensure the correct translation of specified terms, the impact of these methods on translation qual- ity merits closer inspection. Typically, they have been evaluated in terms of general translation met- rics such as BLEU, in addition to the accuracy of the terminology translation. However, there is a dearth of more detailed linguistic analysis of the performance of different techniques; for example, how often do the terms agree morphologically with the rest of the sentence? What are the potential is- sues when unruly, real-world, client glossaries are © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. applied to models trained in more controlled lab- oratory conditions, and what steps can be taken to mitigate these issues? In the present work we implement three ap- proaches to glossary/terminology enforcement in two language pairs (English-Russian and Japanese- English) and compare their performance on the terminology enforcement task. In particular, we investigate two methods based on interventions in the training data and one post-processing method which uses the model’s attention mechanism to identify the tokens representing the translation of the input term in the output and replaces these to- kens with the translation from the glossary. In ad- dition to automated evaluation (chrF, COMET, and accuracy), we also engaged professional linguists to design a test set of edge cases from their partic- ular language pairs, and evaluate the performance of each approach using this bespoke test set. The ultimate objective of this research is to in- form the implementation of a glossary feature for use by machine translation project managers and end users, and thus we must anticipate that the fea- ture will be applied in a multitude of unexpected ways. For a guide to what our feature may be subjected to, we turned to a database of historical glossary enforcement requests kept by our com- pany. These requests were created by a mixture of linguists, clients, and project managers in transla- tion projects. The contents of these glossaries are very noisy and diverse, including nouns, adjectives, …

Content truncated to 500 words