Comparing Multilingual NMT Models and Pivoting

Comparing Multilingual NMT Models and Pivoting Celia Soler Uguet Fred Bane Anna Zaretskaya T`ania Blanch Mir´o TransPerfect {csuguet,fbane,azaretskaya,tblanch}@translations.com Abstract Following recent advancements in multi- lingual machine translation at scale, our team carried out tests to compare the per- formance of pre-trained multilingual mod- els (M2M-100 from Facebook and multi- lingual models from Helsinki-NLP) with a two-step translation process using English as a pivot language. Direct assessment by linguists rated translations produced by pivoting as consistently better than those obtained from multilingual models of sim- ilar size, while automated evaluation with COMET suggested relative performance was strongly impacted by domain and lan- guage family. 1 Background and Motivation As a translation company, our work involves hundreds of distinct translation directions across dozens of languages. However, demand is not evenly distributed across all language pairs. The vast majority of our translation requests involve English as either the source or target language, with most other requests concentrated in a few ma- jor languages, such as German, French, Italian, Japanese, and Chinese. Our fleet of machine translation (MT) engines is developed considering both the demand and the resources available for training. Currently, we use mostly bilingual models with some many-to- one models (such as for Scandinavian languages), but no many-to-many models. For language pairs where only a few hundred words are translated © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. each year, the demand does not justify the costs incurred in training, deploying, and maintaining an engine for that language pair. Moreover, these language pairs often have scant high-quality re- sources available for training. Thus, in situations where demand for machine translation exists, but in insufficient amount to offset training and de- ployment costs, we have historically chosen to use a two-step translation process: pivoting through a related, high-resource language. In recent years, multilingual models have shown growing potential to wholly or partially replace a fleet of bilingual models. The benefits are clear: no error propagation resulting from using the out- put of one model as the input to another as in the pivot scenario; reduced overhead and complexity by using one model for multiple language direc- tions instead of separate models for each direction; improved translation quality in low-resource lan- guages due to knowledge transfer from related lan- guages; the potential for zero-shot translation for language directions for which no direct data ex- ist, and so on. However, these models also have their drawbacks, including the expense and diffi- culty of retraining the models, the inability to add additional languages without retraining the model entirely, and the near impossibility of fine-tuning the model for particular clients. Below we report the results of an experiment comparing bilingual base transformers (Vaswani et al., 2017) with pre-trained M2M-100 from Face- book obtained from Hugging Face (Fan et al., 2020) and multilingual models made public by Helsinki-NLP (Tiedemann and Thottingal, 2020),1 using data drawn from our previous translation work and out-of-domain corpora. 1https://github.com/Helsinki-NLP/Opus-MT 2 Related …

Content truncated to 500 words