Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1246–1257 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1246 Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies Yunsu Kim Yingbo Gao Hermann Ney Human Language Technology and Pattern Recognition Group RWTH Aachen University, Aachen, Germany {surname}@cs.rwth-aachen.de Abstract Transfer learning or multilingual model is essential for low-resource neural machine translation (NMT), but the applicability is limited to cognate languages by sharing their vocabularies. This paper shows effective tech- niques to transfer a pre-trained NMT model to a new, unrelated language without shared vocabularies. We relieve the vocabulary mismatch by using cross-lingual word embed- ding, train a more language-agnostic encoder by injecting artificial noises, and generate synthetic data easily from the pre-training data without back-translation. Our methods do not require restructuring the vocabulary or retraining the model. We improve plain NMT transfer by up to +5.1% BLEU in five low-resource translation tasks, outperforming multilingual joint training by a large margin. We also provide extensive ablation studies on pre-trained embedding, synthetic data, vocabulary size, and parameter freezing for a better understanding of NMT transfer. 1 Introduction Despite recent success of neural machine transla- tion (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017), its major improvements and optimizations cannot be easily applied to low-resource language pairs. Basic training procedure of NMT does not function well with only a handful of bilingual data (Koehn and Knowles, 2017), while collecting bilingual resource is arduous for many languages. Multilingual NMT solves the problem of lack- ing bilingual data by training a shared model along with other related languages (Firat et al., 2016; Johnson et al., 2017). For this to work in prac- tice, however, we need a considerable effort to gather bilingual data over multiple languages and preprocess them jointly before training. This has two critical issues: 1) The languages for training should be linguistically related in order to build a shared vocabulary. 2) It is not feasible to add a new language to a trained model, since the train- ing vocabulary must be redefined; one may need to re-train the model from scratch. In transfer learning (Zoph et al., 2016), adapting to a new language is conceptually simpler; given an NMT model pre-trained on a high-resource lan- guage pair (parent), we can just continue the train- ing with bilingual data of another language pair (child). Here, the vocabulary mismatch between languages is still a problem, which seriously limits the performance especially for distant languages. This work proposes three novel ideas to make transfer learning for NMT widely applicable to various languages: • We alleviate the vocabulary mismatch be- tween parent and child languages via cross- lingual word embedding. • We train a more general encoder in the parent training by injecting artificial noises, making it easier for the child model to adapt to. • We generate synthetic data from parallel data of the parent language pair, improv- ing the low-resource transfer where the …

Content truncated to 500 words