Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages

Proceedings of NAACL-HLT 2019, pages 3868–3873 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3868 Addressing Word-order Divergence in Multilingual Neural Machine Translation for Extremely Low Resource Languages Rudra Murthy V†, Anoop Kunchukuttan‡, Pushpak Bhattacharyya† † Center for Indian Language Technology (CFILT) Department of Computer Science and Engineering IIT Bombay, India. ‡Microsoft AI & Research, Hyderabad, India. {rudra,pb}@cse.iitb.ac.in, ankunchu@microsoft.com Abstract Transfer learning approaches for Neural Ma- chine Translation (NMT) trains a NMT model on an assisting language-target language pair (parent model) which is later fine-tuned for the source language-target language pair of in- terest (child model), with the target language being the same. In many cases, the assisting language has a different word order from the source language. We show that divergent word order adversely limits the benefits from trans- fer learning when little to no parallel corpus be- tween the source and target language is avail- able. To bridge this divergence, we propose to pre-order the assisting language sentences to match the word order of the source language and train the parent model. Our experiments on many language pairs show that bridging the word order gap leads to major improve- ments in the translation quality in extremely low-resource scenarios. 1 Introduction Transfer learning for multilingual Neural Machine Translation (NMT) (Zoph et al., 2016; Dabre et al., 2017; Nguyen and Chiang, 2017) attempts to im- prove the NMT performance on the source to tar- get language pair (child task) using an assisting source language (assisting to target language trans- lation is the parent task). Here, the parent model is trained on the assisting and target language paral- lel corpus and the trained weights are used to ini- tialize the child model. If source-target language pair parallel corpus is available, the child model can further be fine-tuned. The weight initialization reduces the requirement on the training data for the source-target language pair by transferring knowl- edge from the parent task, thereby improving the performance on the child task. However, the divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning. Mul- tiple studies have shown that transfer learning works best when the languages are related (Zoph et al., 2016; Nguyen and Chiang, 2017; Dabre et al., 2017). Zoph et al. (2016) studied the in- fluence of language divergence between languages chosen for training the parent and the child model, and showed that choosing similar languages for training the parent and the child model leads to bet- ter improvements from transfer learning. Several studies have tried to address the lex- ical divergence between the source and the tar- get languages either by using Byte Pair Encoding (BPE) as basic input representation units (Nguyen and Chiang, 2017) or character-level NMT sys- tem (Lee et al., 2017) or bilingual embeddings (Gu et al., 2018). However, the effect of word order di- vergence and its mitigation has not been explored. In a practical setting, it is not uncommon to have source and assisting …

Content truncated to 500 words