An Empirical Study of Language Relatedness for Transfer Learning in Neural Machine Translation

31st Pacific Asia Conference on Language, Information and Computation (PACLIC 31), pages 282–286 Cebu City, Philippines, November 16-18, 2017 Copyright c⃝2017 Raj Dabre, Tetsuji Nakagawa and Hideto Kazawa An Empirical Study of Language Relatedness for Transfer Learning in Neural Machine Translation Raj Dabre Kyoto University, Kyoto, Japan prajdabre@gmail.com Tetsuji Nakagawa Google Japan, Tokyo, Japan tnaka@google.com Hideto Kazawa Google Japan, Tokyo, Japan kazawa@google.com Abstract Neural Machine Translation (NMT) is known to outperform Phrase Based Sta- tistical Machine Translation (PBSMT) for resource rich language pairs but not for resource poor ones. Transfer Learning (Zoph et al., 2016) is a simple approach in which we can simply initialize an NMT model (child model) for a resource poor language pair using a previously trained model (parent model) for a resource rich language pair where the target languages are the same. This paper explores how dif- ferent choices of parent models affect the performance of child models. We empiri- cally show that using a parent model with the source language falling in the same or linguistically similar language family as the source language of the child model is the best. 1 Introduction One of the most attractive features of Neural Ma- chine Translation (NMT) (Bahdanau et al., 2015; Cho et al., 2014; Sutskever et al., 2014) is that it is possible to train an end to end system without the need to deal with word alignments, phrase ta- bles and complicated decoding algorithms which are a characteristic of Phrase Based Statistical Ma- chine Translation (PBSMT) systems (Koehn et al., 2003). It is reported that NMT works better than PBSMT only when there is an abundance of paral- lel corpora. In the case of low resource languages like Hausa, vanilla NMT is either worse than or comparable to PBSMT (Zoph et al., 2016). How- ever, it is possible to use a previously trained X-Y model (parent model; X-Y being the resource rich language pair where X and Y represent the source and target languages respectively) to initialize the parameters of a Z-Y model (child model; Z-Y being the resource poor language pair) leading to significant improvements (Zoph et al., 2016) for the latter. This paper is about an empirical study of transfer learning for NMT for low re- source languages. Our main focus is on trans- lation to English for the following low resource languages: Hausa, Uzbek, Marathi, Malayalam, Punjabi, Malayalam, Kazakh, Luxembourgish, Ja- vanese and Sundanese. Our main contribution is that we empirically (and exhaustively; within reason) show that using a resource rich language pair in which the source language is linguistically closer to the source language of the resource poor pair is much better than other choices of language pairs. 2 Related Work Transfer learning for NMT (Zoph et al., 2016) is an approach where previously trained NMT mod- els for French and German to English (resource rich pairs) were used to initialize models for Hausa, Uzbek, Spanish to English (resource poor pairs). They showed that French-English as a par- ent model was better …

Content truncated to 500 words