Importance-based Neuron Allocation for Multilingual Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5725–5737 August 1–6, 2021. ©2021 Association for Computational Linguistics 5725 Importance-based Neuron Allocation for Multilingual Neural Machine Translation Wanying Xie1,2,3 Yang Feng1,2∗ Shuhao Gu1,2 Dong Yu3 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China 3 Beijing Language and Culture University, China xiewanying07@gmail.com, yudong@blcu.edu.cn {fengyang, gushuhao19b}@ ict.ac.cn Abstract Multilingual neural machine translation with a single model has drawn much attention due to its capability to deal with multiple languages. However, the current multilingual translation paradigm often makes the model tend to pre- serve the general knowledge, but ignore the language-specific knowledge. Some previous works try to solve this problem by adding var- ious kinds of language-specific modules to the model, but they suffer from the parameter ex- plosion problem and require specialized man- ual design. To solve these problems, we pro- pose to divide the model neurons into general and language-specific parts based on their im- portance across languages. The general part is responsible for preserving the general knowl- edge and participating in the translation of all the languages, while the language-specific part is responsible for preserving the language- specific knowledge and participating in the translation of some specific languages. Ex- perimental results on several language pairs, covering IWSLT and Europarl corpus datasets, demonstrate the effectiveness and universality of the proposed method. 1 Introduction Neural machine translation(NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) has shown its superiority and drawn much attention in recent years. Although the NMT model can achieve promising results for high- resource language pairs, it is unaffordable to train separate models for all the language pairs since there are thousands of languages in the world (Tan et al., 2019; Aharoni et al., 2019; Arivazhagan et al., 2019). A typical solution to reduce the model size ∗Corresponding author: Yang Feng. Our code can be got at https://github.com/ictnlp/NA- MNMT and the training cost is to handle multiple languages in a single multilingual neural machine translation (MNMT) model (Ha et al., 2016; Firat et al., 2016; Johnson et al., 2017; Gu et al., 2018). The standard paradigm of MNMT proposed by Johnson et al. (2017) contains a language-shared encoder and de- coder with a special language indicator in the input sentence to determine the target language. Because different languages share all of the model parameters in the standard MNMT model, the model tends to converge to a region where there are low errors for all the languages. Therefore, the MNMT model trained on the combined data generally captures the general knowledge, but ig- nores the language-specific knowledge, rendering itself sub-optimal for the translation of a specific language (Sachan and Neubig, 2018; Blackwood et al., 2018; Wang et al., 2020b). To retain the language-specific knowledge, some researches …

Content truncated to 500 words