Language Tokens: Simply Improving Zero-Shot Multi-Aligned Translation in Encoder-Decoder Models

Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translation Muhammad ElNokrashy muelnokr@microsoft.com Amr Hendy amrhendy@microsoft.com Mohamed Maher mohamedmaher@microsoft.com Mohamed Afify mafify@microsoft.com Microsoft ATL, Cairo Hany Hassan Awadalla hanyh@microsoft.com Microsoft, Redmond Abstract This paper proposes a simple yet effective method to improve direct (X-to-Y) translation for both cases: zero-shot and when direct data is available. We modify the input tokens at both the encoder and decoder to include signals for the source and target languages. We show a performance gain when training from scratch, or finetuning a pretrained model with the pro- posed setup. In the experiments, our method shows nearly 10.0 BLEU points gain on in-house datasets depending on the checkpoint selection criteria. In a WMT evaluation campaign, From- English performance improves by 4.17 and 2.87 BLEU points, in the zero-shot setting, and when direct data is available for training, respectively. While X-to-Y improves by 1.29 BLEU over the zero-shot baseline, and 0.44 over the many-to-many baseline. In the low-resource setting, we see a 1.5 ∼1.7 point improvement when finetuning on X-to-Y domain data. 1 Introduction Neural machine translation (NMT) has witnessed significant advances since the introduction of the transformer model (Vaswani et al., 2017). This model has shown impressive performance for bilingual translation commonly from and to English (Hassan et al., 2018). It has also been shown that the proposed model could be easily extended to multiple language pairs (Aharoni, Johnson, & Firat, 2019; Fan et al., 2020; Johnson et al., 2017; X. Wang, Tsvetkov, & Neubig, 2020), to and/or from English, by simple modifications to the basic architecture. This holds promise for improved performance for low-resource pairs through transfer learning, as well as better training and deployment costs per language pair. This setting is referred to as multilingual neural machine translation (MNMT). The mainstream method of training MNMT is to introduce an additional input tag at the encoder to indicate the target language, while the decoder uses the usual begin-of-sentence ( BOS ) token. This simple modification to the bilingual architecture is shown to work well up to hundreds of language pairs (Fan et al., 2020; Tran et al., 2021), given a corresponding in- crease in the number of parameters to handle the increased training data. Despite the emergence of modified architectures which add language-specific parameters, like language specific sub- networks (LASS) (Lin, Wu, Wang, & Li, 2021), and adapters (Bapna & Firat, 2019), the basic architecture remains the most effective choice for deploying large scale production systems. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 70 SRC TGT TGT Encoder Tgt BOS (Baseline) Src,Tgt Tgt TGT BOS Decoder Figure 1: Comparing tokens as seen by the Encoder and the Decoder in the (Left) baseline ( T-B ) and in the (Right) top proposed method ( ST-T ). 2 Motivation While MNMT was originally focused on English-centric translation, there is increasing interest interest in direct translation1 rather than pivoting …

Content truncated to 500 words