Transfer Learning for Low-Resource Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Transfer Learning for Low-Resource Neural Machine Translation Barret Zoph* Deniz Yuret Jonathan May and Kevin Knight Information Sciences Institute Computer Engineering Information Sciences Institute University of Southern California Koc¸ University Computer Science Department barretzoph@gmail.com dyuret@ku.edu.tr University of Southern California {jonmay,knight}@isi.edu Abstract The encoder-decoder framework for neural machine translation (NMT) has been shown effective in large data scenarios, but is much less effective for low-resource languages. We present a transfer learning method that signifi- cantly improves BLEU scores across a range of low-resource languages. Our key idea is to first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. Using our transfer learning method we improve baseline NMT models by an av- erage of 5.6 BLEU on four low-resource lan- guage pairs. Ensembling and unknown word replacement add another 2 BLEU which brings the NMT performance on low-resource ma- chine translation close to a strong syntax based machine translation (SBMT) system, exceed- ing its performance on one language pair. Ad- ditionally, using the transfer learning model for re-scoring, we can improve the SBMT sys- tem by an average of 1.3 BLEU, improving the state-of-the-art on low-resource machine translation. 1 Introduction Neural machine translation (NMT) (Sutskever et al., 2014) is a promising paradigm for extracting trans- lation knowledge from parallel text. NMT sys- tems have achieved competitive accuracy rates un- der large-data training conditions for language pairs This work was carried out while all authors were at USC’s Information Sciences Institute. *This author is currently at Google Brain. Language Train Test SBMT NMT Size Size BLEU BLEU Hausa 1.0m 11.3K 23.7 16.8 Turkish 1.4m 11.6K 20.4 11.4 Uzbek 1.8m 11.5K 17.9 10.7 Urdu 0.2m 11.4K 17.9 5.2 Table 1: NMT models with attention are outperformed by stan- dard string-to-tree statistical MT (SBMT) when translating low- resource languages into English. Train/test bitext corpus sizes are English token counts. Single-reference, case-insensitive BLEU scores are given for held-out test corpora. such as English–French. However, neural methods are data-hungry and learn poorly from low-count events. This behavior makes vanilla NMT a poor choice for low-resource languages, where parallel data is scarce. Table 1 shows that for 4 low-resource languages, a standard string-to-tree statistical MT system (SBMT) (Galley et al., 2004; Galley et al., 2006) strongly outperforms NMT, even when NMT uses the state-of-the-art local attention plus feed- input techniques from Luong et al. (2015a). In this paper, we describe a method for substan- tially improving NMT results on these languages. Our key idea is to first train a high-resource lan- guage pair, then use the resulting trained network (the parent model) to initialize and constrain training for our low-resource language pair (the child model). We find that we can optimize our results by fixing certain parameters of the parent model and letting the rest …

Content truncated to 500 words