Multilingual Neural Machine Translation With the Right Amount of Sharing

Multilingual Neural Machine Translation With the Right Amount of Sharing Taido Purason University of Tartu Tartu, Estonia taido.purason@ut.ee Andre T¨attar University of Tartu Tartu, Estonia andre.tattar@ut.ee Abstract Large multilingual Transformer-based ma- chine translation models have had a pivotal role in making translation systems avail- able for hundreds of languages with good zero-shot translation performance. One such example is the universal model with shared encoder-decoder architecture. Ad- ditionally, jointly trained language-specific encoder-decoder systems have been pro- posed for multilingual neural machine translation (NMT) models. This work in- vestigates various knowledge-sharing ap- proaches on the encoder side while keep- ing the decoder language- or language- group-specific. We propose a novel ap- proach, where we use universal, language- group-specific and language-specific mod- ules to solve the shortcomings of both the universal models and models with language-specific encoders-decoders. Ex- periments on a multilingual dataset set up to model real-world scenarios, includ- ing zero-shot and low-resource translation, show that our proposed models achieve higher translation quality compared to purely universal and language-specific ap- proaches. 1 Introduction Multilingual neural machine translation has been a fundamental topic in recent years, especially for zero- and few-shot translation scenarios. Tradi- tionally, universal NMT models (see Fig. 1a) have © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. Univ. Univ. tgt lang + (a) Universal model gem gem roa roa tgt lang + (b) Model with language-group- specific modules en de es en es de (c) Model with language- specific modules Figure 1: Different granularities of the modular architecture. roa – Romance; gem – Germanic; tgt lang – Target language token added to indicate the language of the output sentence. been used to produce zero-shot or low-resource translations (Johnson et al., 2016). However, pre- vious research has established that universal NMT models with shared encoder-decoder architecture have some disadvantages: (1) high-resource lan- guage pairs tend to suffer loss in translation qual- ity (Arivazhagan et al., 2019); (2) the vocabulary of the model increases greatly, especially for lan- guages that do not share an alphabet such as En- glish and Japanese; (3) the need to retrain from scratch when a new language does not share the model’s vocabulary. Recently, there has been renewed interest in multilingual systems, which have jointly trained language-specific encoders-decoders (see Fig. 1c) which we call the modular architecture (Lyu et al., 2020). The goal of these models has been to achieve a better overall translation quality com- pared to universal or uni-directional NMT models. However, there is a disadvantage: lower zero-shot translation quality compared to universal models. To combat this problem, shared encoder/decoder layers (also called interlingua layers) have been proposed (Liao et al., 2021). en G de fr es R en es fr de (a) Language group sharing en G U fr es R en es fr de de (b) groupLanguage and universal sharing (tiered) en de fr es en es fr de U (c) Universal sharing (between all languages) Figure 2: Different types of …

Content truncated to 500 words