Language Tags Matter for Zero-Shot Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001–3007 August 1–6, 2021. ©2021 Association for Computational Linguistics 3001 Language Tags Matter for Zero-Shot Neural Machine Translation Liwei Wu, Shanbo Cheng, Mingxuan Wang, Lei Li ByteDance AI Lab {wuliwei.000,chenshanbo,wangmingxuan.89,lileilab}@bytedance.com Abstract Multilingual Neural Machine Translation (MNMT) has aroused widespread interest due to its efficiency. An exciting advantage of MNMT models is that they could also translate between unsupervised (zero-shot) language directions. Language tag (LT) strategies are often adopted to indicate the translation directions in MNMT. In this paper, we demonstrate that the LTs are not only indicators for translation directions but also crucial to zero-shot translation qualities. Un- fortunately, previous work tends to ignore the importance of LT strategies. We demonstrate that a proper LT strategy could enhance the consistency of semantic representations and alleviate the off-target issue in zero-shot directions. Experimental results show that by ignoring the source language tag (SLT) and adding the target language tag (TLT) to the encoder, the zero-shot translations could achieve a +8 BLEU score difference over other LT strategies in IWSLT17, Europarl, TED talks translation tasks. 1 Introduction Neural Machine Translation (NMT) based on the encoder-decoder framework with attention mecha- nism (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017) has achieved state-of- the-art (SotA) results in many language pairs (Deng et al., 2018; Barrault et al., 2019). Pioneered by (Dong et al., 2015; Firat et al., 2016; Zoph and Knight, 2016; Ha et al., 2016; Johnson et al., 2017), researchers start to investigate the possibility of using a single model to translate between multi- ple languages, which is the Multilingual Neural Machine Translation (MNMT). Benefiting from the transferring ability of multilingual modeling, MNMT could achieve better translation quality be- tween low-resource language directions than bilin- gual models (Gu et al., 2018; Wang et al., 2019). More exciting, MNMT could even translate be- tween zero-shot language directions (Johnson et al., 2017; Gu et al., 2019; Pham et al., 2019; Kudugunta et al., 2019). Unlike bilingual NMT, language-specific sig- nals should be accessible to the MNMT model so that the model can distinguish the translation direc- tions. Ha et al., (2016) first introduced a universal encoder-decoder framework for MNMT models with language-specific coded vocabulary to indi- cate different languages. The encoder-decoder ar- chitecture is identical to bilingual models (Bah- danau et al., 2015; Vaswani et al., 2017). To further simplify the MNMT models, Johnson et al., (2017) propose to add language tags (LTs) to the begin- ning of input data to indicate the target language. Then a shared vocabulary could be learned for all languages. The training data of different languages could thus be mixed-up to train the MNMT model. Such a strategy greatly simplifies the training and decoding procedure. We call it the LT strategy. This paper focuses on investigating the impact of LT strategies for zero-shot translation directions in MNMT (zero-shot MNMT). We conduct trans- lation experiments …

Content truncated to 500 words