A Compact and Language-Sensitive Multilingual Translation Method

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1213–1223 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1213 A Compact and Language-Sensitive Multilingual Translation Method Yining Wang1,2, Long Zhou1,2, Jiajun Zhang1,2∗, Feifei Zhai4, Jingfang Xu4 and Chengqing Zong1,2,3 1National Laboratory of Pattern Recognition, CASIA, Beijing, China 2University of Chinese Academy of Sciences, Beijing, China 3CAS Center for Excellence in Brain Science and Intelligence Technology, Beijing, China 4Sogou Inc., Beijing, China {yining.wang, long.zhou, jjzhang, cqzong}@nlpr.ia.ac.cn {zhaifeifei,xujingfang}@sogou-inc.com Abstract Multilingual neural machine translation (Multi-NMT) with one encoder-decoder model has made remarkable progress due to its simple deployment. However, this multi- lingual translation paradigm does not make full use of language commonality and param- eter sharing between encoder and decoder. Furthermore, this kind of paradigm cannot outperform the individual models trained on bilingual corpus in most cases. In this paper, we propose a compact and language-sensitive method for multilingual translation. To maximize parameter sharing, we first present a universal representor to replace both encoder and decoder models. To make the representor sensitive for specific languages, we further introduce language-sensitive embedding, attention, and discriminator with the ability to enhance model performance. We verify our methods on various translation scenarios, including one-to-many, many-to-many and zero-shot. Extensive experiments demon- strate that our proposed methods remarkably outperform strong standard multilingual translation systems on WMT and IWSLT datasets. Moreover, we find that our model is especially helpful in low-resource and zero-shot translation scenarios. 1 Introduction Encoder-decoder based sequence-to-sequence ar- chitecture (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Zhang and Zong, 2015; Vaswani et al., 2017; Gehring et al., 2017) fa- cilitates the development of multilingual neural machine translation (Multi-NMT) (Dong et al., 2015; Luong et al., 2016; Firat et al., 2016; John- son et al., 2017; Gu et al., 2018). The domi- ∗ Jiajun Zhang is the corresponding author and the work is done while Yining Wang is doing research intern at Sogou Inc.  N  La ng-S i ens Emb ed ding La i ng-Sens at tent ion Discrimn at or M Sr c N Tg t Enc oder Deco der Shar ing Rep res en to r N Lang-Sensi Embedding Lang-Sensi Attention Lang-Sensi Discrimnator M Src N Tgt Encoder Decoder Sharing Representor Figure 1: Our proposed compact representor, replacing encoder and decoder, can perform multilingual trans- lation from M source languages to N target languages. We also introduce three specific modules consisting of language-sensitive embedding, language-sensitive at- tention, and language-sensitive discriminator. nant paradigm of Multi-NMT contains one en- coder to represent multiple languages and one de- coder to generate output tokens of separate lan- guages (Johnson et al., 2017; Ha et al., 2016). This paradigm is widely used in Multi-NMT systems due to simple implementation and convenient de- ployment. However, this paradigm has two drawbacks. For one hand, using single encoder-decoder frame- work for all language pairs usually yields infe- rior performance compared to individually trained single-pair models in most cases (Lu et al., 2018; …

Content truncated to 500 words