Multilingual Neural Machine Translation with Language Clustering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 963–973, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 963 Multilingual Neural Machine Translation with Language Clustering Xu Tan§∗, Jiale Chen†∗, Di He‡, Yingce Xia§, Tao Qin§ and Tie-Yan Liu§ §Microsoft Research †University of Science and Technology of China ‡Peking University {xuta,taoqin,tyliu}@microsoft.com Abstract Multilingual neural machine translation (NMT), which translates multiple languages using a single model, is of great practi- cal importance due to its advantages in simplifying the training process, reducing online maintenance costs, and enhancing low-resource and zero-shot translation. Given there are thousands of languages in the world and some of them are very different, it is extremely burdensome to handle them all in a single model or use a separate model for each language pair. Therefore, given a fixed resource budget, e.g., the number of models, how to determine which languages should be supported by one model is critical to multilingual NMT, which, unfortunately, has been ignored by previous work. In this work, we develop a framework that clusters languages into different groups and trains one multilingual model for each cluster. We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space. In particular, we obtain the embedding vectors of all the languages by training a universal neural machine translation model. Our experiments on 23 languages show that the first clustering method is simple and easy to understand but leading to suboptimal translation accuracy, while the second method sufficiently captures the relationship among languages well and improves the translation accuracy for almost all the languages over baseline methods. 1 Introduction Neural machine translation (NMT) (Bahdanau et al., 2015; Luong et al., 2015b; Sutskever et al., ∗Authors contribute equally to this work. 2014; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017) has witnessed rapid progress in recent years, from novel model structure de- velopments (Gehring et al., 2017; Vaswani et al., 2017) to achieving performance comparable to hu- mans (Hassan et al., 2018). Although a conventional NMT model can handle a single language translation pair (e.g., German→English, Spanish→French) well, train- ing a separate model for each language pair is un- affordable considering there are thousands of lan- guages in the world. A straightforward solution to reduce computational cost is using one model to handle the translations of multiple languages, i.e., multilingual translation. Johnson et al. (2017); Fi- rat et al. (2016); Ha et al. (2016); Lu et al. (2018) propose to share part of (e.g., attention mecha- nism) or all models for multiple language pairs and achieve considerable accuracy improvement. While they focus on how to translate multiple lan- guage pairs in a single model and improve the per- formance of the multilingual model, they do not investigate …

Content truncated to 500 words