Adapting to Non-Centered Languages for Zero-shot Multilingual Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5251–5265 October 12–17, 2022. 5251 Adapting to Non-Centered Languages for Zero-shot Multilingual Translation Zhi Qu and Taro Watanabe Nara Institute of Science and Technology {qu.zhi.pv5, taro}@is.naist.jp Abstract Multilingual neural machine translation can translate unseen language pairs during training, i.e. zero-shot translation. However, the zero- shot translation is always unstable. Although prior works attributed the instability to the dom- ination of central language, e.g. English, we supplement this viewpoint with the strict de- pendence of non-centered languages. In this work, we propose a simple, lightweight yet ef- fective language-specific modeling method by adapting to non-centered languages and com- bining the shared information and the language- specific information to counteract the instabil- ity of zero-shot translation. Experiments with Transformer on IWSLT17, Europarl, TED talks, and OPUS-100 datasets show that our method not only performs better than strong baselines in centered data conditions but also can easily fit non-centered data conditions. By further in- vestigating the layer attribution, we show that our proposed method can disentangle the cou- pled representation in the correct direction.1 1 Introduction Training multilingual neural machine translation (MNMT) system requires enormous number of pa- rameters and resources, but the zero-shot transla- tion, namely translating unseen language pairs dur- ing training, has shown the potential to simplify the MNMT (Firat et al., 2017). Johnson et al. (2017) has shown that adding language tokens, e.g. <en>, at the beginning of a sentence allows the model to build cross-linguistic representation by treating the token as translation instruction specifying target language. However, the zero-shot translation is al- ways unstable. One possibility causing the instabil- ity of zero-shot translation is spurious correlation (Gu et al., 2019). The target linguistic representa- tion captured by the model is directly and strictly 1Codes and detailed results are available in: https:// github.com/zhiqu22/AdapNonCenter dependent on encoded source linguistic informa- tion instead of learning specific representations for source and target language, then combining inde- pendent linguistic representations to generate re- sults. Prior works (Lakew et al., 2019; Fan et al., 2020; Rios et al., 2020; Freitag and Firat, 2020; Liu et al., 2021) indicated that the spurious correlation is caused by the centered data condition in which multilingual data is constructed by bridging a cen- tral language, e.g. English, to other non-centered languages. The central language will dominate the representation in the MNMT model to degenerate the information specific to non-centered languages since multilingual data comprises a set of bilin- gual data constructed by coupling non-centered languages with the central language. However, the non-centered data condition without any central language is also unstable in zero-shot translation.2 Therefore, simply attributing the instability of zero- shot translation to the central language cannot fit all cases of zero-shot translation. We move the perspective from the domination of the central language to the weakness of non- centered languages. The problems of zero-shot translation could be attributed to the strict depen- dence of non-centered languages. Specifically, a non-centered language would strictly depend on an- other …

Content truncated to 500 words