Adaptive Token-level Cross-lingual Feature Mixing for Multilingual Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10097 - 10113 December 7-11, 2022 ©2022 Association for Computational Linguistics Adaptive Token-level Cross-lingual Feature Mixing for Multilingual Neural Machine Translation Junpeng Liu1 Kaiyu Huang2 Jiuyi Li1 Huan Liu1 Jinsong Su3 Degen Huang1∗ 1Dalian University of Technology 2Institute for AI Industry Research, Tsinghua University 3Xiamen University {liujunpeng_nlp, lee.91, liuhuan4221}@mail.dlut.edu.cn huangkaiyu@air.tsinghua.edu.cn jssu@xmu.edu.cn huangdg@dlut.edu.cn Abstract Multilingual neural machine translation aims to translate multiple language pairs in a sin- gle model and has shown great success thanks to the knowledge transfer across languages with the shared parameters. Despite promis- ing, this share-all paradigm suffers from in- sufficient ability to capture language-specific features. Currently, the common practice is to insert or search language-specific networks to balance the shared and specific features. How- ever, those two types of features are not suffi- cient enough to model the complex common- ality and divergence across languages, such as the locally shared features among similar lan- guages, which leads to sub-optimal transfer, especially in massively multilingual translation. In this paper, we propose a novel token-level feature mixing method that enables the model to capture different features and dynamically determine the feature sharing across languages. Based on the observation that the tokens in the multilingual model are usually shared by different languages, we insert a feature mix- ing layer into each Transformer sublayer and model each token representation as a mix of dif- ferent features, with a proportion indicating its feature preference. In this way, we can perform fine-grained feature sharing and achieve bet- ter multilingual transfer. Experimental results on multilingual datasets show that our method outperforms various strong baselines and can be extended to zero-shot translation. Further analyses reveal that our method can capture different linguistic features and bridge the rep- resentation gap across languages.1 1 Introduction Multilingual neural machine translation (MNMT) (Ha et al., 2016; Johnson et al., 2017) handles sev- eral translation directions in a single model. These ∗Corresponding Author 1Our code is available at https://github.com/ raburabu91/HiTrans multilingual models have been shown to be capa- ble of facilitating the knowledge transfer across different languages (Lakew et al., 2018; Tan et al., 2019; Zhang et al., 2020) and enabling translations between language pairs unseen in training (John- son et al., 2017; Al-Shedivat and Parikh, 2019; Gu et al., 2019; Zhang et al., 2020). Due to the above advantages, MNMT is appealing and has drawn much attention in recent years. The success of MNMT comes at the cost of insuf- ficient ability to capture language-specific features (Zhang et al., 2021). Since the model parameters are shared across languages, the MNMT model tends to preserve the shared features but ignore the language-specific ones. Therefore, researchers re- sort to language-specific modeling to capture and balance those two types of features. Some works attempt to insert additional language-specific mod- ules into the original MNMT model (Wang et al., 2019; Bapna and Firat, 2019; Zhang et al., 2020, 2021). However, those methods are sensitive to the structure and location …

Content truncated to 500 words