Informative Language Representation Learning for Massively Multilingual Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5158–5174 October 12–17, 2022. 5158 Informative Language Representation Learning for Massively Multilingual Neural Machine Translation Renren Jin and Deyi Xiong ∗ College of Intelligence and Computing, Tianjin University, Tianjin, China {rrjin, dyxiong}@tju.edu.cn Abstract In a multilingual neural machine translation model that fully shares parameters across all languages, an artificial language token is usu- ally used to guide translation into the desired target language. However, recent studies show that prepending language tokens sometimes fails to navigate the multilingual neural ma- chine translation models into right translation directions, especially on zero-shot translation. To mitigate this issue, we propose two meth- ods, language embedding embodiment and language-aware multi-head attention, to learn informative language representations to chan- nel translation into right directions. The former embodies language embeddings into different critical switching points along the information flow from the source to the target, aiming at am- plifying translation direction guiding signals. The latter exploits a matrix, instead of a vec- tor, to represent a language in the continuous space. The matrix is chunked into multiple heads so as to learn language representations in multiple subspaces. Experiment results on two datasets for massively multilingual neural machine translation demonstrate that language- aware multi-head attention benefits both super- vised and zero-shot translation and significantly alleviates the off-target translation issue. Fur- ther linguistic typology prediction experiments show that matrix-based language representa- tions learned by our methods are capable of capturing rich linguistic typology features.1 1 Introduction Multilingual neural machine translation (MNMT) (Johnson et al., 2017; Ha et al., 2016; Aharoni et al., 2019; Arivazhagan et al., 2019b), unlike bilingual machine translation with task-specific engineering (language-specific features), uses a single learning system for multiple language pairs, which is jointly ∗Corresponding author. 1The source code is publicly available at https:// github.com/cordercorder/nmt-multi. trained in a mult-task learning formalism (Col- lobert et al., 2011). Parameter sharing across dif- ferent languages in MNMT models enables trans- ferring of intermediate representations and knowl- edge among languages, which makes it beneficial to machine translation of low-resource and even zero-resource languages (Firat et al., 2016b; Gu et al., 2018; Neubig and Hu, 2018; Gu et al., 2019; Zhang et al., 2020). According to the degree of parameter sharing across languages, multilingual neural machine translation (MNMT) approaches can be categorized into two strands: full parameter sharing and partial parameter sharing (Sachan and Neubig, 2018). The former uses a unified model where both the encoder and decoder are shared for all languages. To guide the translation direction, a special token is usually prepended to the beginning of the source or target sentence to indicate the target language (Johnson et al., 2017; Fan et al., 2021). Instead of sharing all model parameters for all translation directions, the latter uses language-specific components, e.g., separate encoders (in many-to-one translation), sep- arate decoders (in one-to-many translation), or sep- arate cross-attention networks (Zoph and Knight, 2016; Firat et al., 2016a; Blackwood et al., 2018; Vázquez et al., 2020; Kong et al., 2021). The key …

Content truncated to 500 words