Language-aware Interlingua for Multilingual Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1650–1655 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1650 Language-aware Interlingua for Multilingual Neural Machine Translation Changfeng Zhu, Heng Yu, Shanbo Cheng, Weihua Luo Machine Intelligence Technology Lab, Alibaba Group {changfeng.zcf,yuheng.yh,shanbo.csb,weihua.luowh} @alibaba-inc.com Abstract Multilingual neural machine translation (NMT) has led to impressive accuracy im- provements in low-resource scenarios by sharing common linguistic information across languages. However, the traditional multilin- gual model fails to capture the diversity and specificity of different languages, resulting in inferior performance compared with indi- vidual models that are sufficiently trained. In this paper, we incorporate a language-aware interlingua into the Encoder-Decoder archi- tecture. The interlingual network enables the model to learn a language-independent representation from the semantic spaces of different languages, while still allowing for language-specific specialization of a particular language-pair. Experiments show that our proposed method achieves remarkable im- provements over state-of-the-art multilingual NMT baselines and produces comparable performance with strong individual models. 1 Introduction Neural Machine Translation (NMT) (Sutskever et al., 2014; Vaswani et al., 2017) has signifi- cantly improved the translation quality due to its end-to-end modeling and continuous representa- tion. While conventional NMT performs single pair translation well, training a separate model for each language pair is resource consuming, con- sidering there are thousands of languages in the world. Therefore multilingual NMT is introduced to handle multiple language pairs in one model, reducing the online serving and offline training cost. Furthermore, the multilingual NMT frame- work facilitates the cross-lingual knowledge trans- fer to improve translation performance on low re- source language pairs (Wang et al., 2019). Despite all the mentioned advantages, multi- lingual NMT remains a challenging task since the language diversity and model capacity limita- tions lead to inferior performance against individ- ual models that are sufficiently trained. So recent efforts in multilingual NMT mainly focus on en- larging the model capacity, either by introducing multiple Encoders and Decoders to handle differ- ent languages (Firat et al., 2016; Zoph and Knight, 2016), or enhancing the attention mechanism with language-specific signals (Blackwood et al., 2018). On the other hand, there have been some efforts to model the specificity of different lan- guages. Johnson et al. (2017) and Ha et al. (2016) tackle this by simply adding some pre-designed tokens at the beginning of the source/target se- quence, but we argue that such signals are not strong enough to learn enough language-specific information to transform the continuous represen- tation of each language into the shared semantic space based on our observations. In this paper, we incorporate a language-aware Interlingua module into the Encoder-Decoder ar- chitecture. It explicitly models the shared seman- tic space for all languages and acts as a bridge be- tween the Encoder and Decoder network. Specif- ically, we first introduce a language embedding to represent unique characteristics of each language and an interlingua embedding to capture the com- mon semantics across languages. Then we use the two embeddings to augment the self-attention mechanism which …

Content truncated to 500 words