Trivial Transfer Learning for Low-Resource Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 244–252 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64025 Trivial Transfer Learning for Low-Resource Neural Machine Translation Tom Kocmi Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic <surname>@ufal.mff.cuni.cz Abstract Transfer learning has been proven as an ef- fective technique for neural machine transla- tion under low-resource conditions. Exist- ing methods require a common target lan- guage, language relatedness, or specific train- ing tricks and regimes. We present a simple transfer learning method, where we first train a “parent” model for a high-resource language pair and then continue the training on a low- resource pair only by replacing the training corpus. This “child” model performs signifi- cantly better than the baseline trained for low- resource pair only. We are the first to show this for targeting different languages, and we observe the improvements even for unrelated languages with different alphabets. 1 Introduction Neural machine translation (NMT) has made a big leap in performance and became the unquestion- able winning approach in the past few years (Bah- danau et al., 2014; Sutskever et al., 2014; Sennrich et al., 2017; Vaswani et al., 2017). The main rea- son behind the success of NMT in realistic con- ditions was the ability to handle large vocabulary (Sennrich et al., 2016b) and to utilize large mono- lingual data (Sennrich et al., 2016a). However, NMT still struggles if the parallel data is insuffi- cient (e.g. fewer than 1M parallel sentences), pro- ducing fluent output unrelated to the source and performing much worse than phrase-based ma- chine translation (Koehn and Knowles, 2017). Many strategies have been used in MT in the past for employing resources from additional lan- guages, see e.g. Wu and Wang (2007), Nakov and Ng (2012), El Kholy et al. (2013), or Hoang and Bojar (2016). For NMT, a particularly promising approach is transfer learning or “domain adapta- tion” where the “domains” are the different lan- guages. For example, Zoph et al. (2016) train a “par- ent” model in a high-resource language pair, then use some of the trained weights as the initializa- tion for a “child” model and further train it on the low-resource language pair. In Zoph et al. (2016), the parent and child pairs shared the target lan- guage (English) and a number of modifications of the training process were needed to achieve an improvement in translation from Hansa, Turkish, and Uzbek into English with the help of French- English data. Nguyen and Chiang (2017) explore a related scenario where the parent language pair is also low-resource but it is related to the child language pair. They improved the previous approach by us- ing a shared vocabulary of subword units (BPE, Sennrich et al., 2016b). Additionally, they used transliteration to improve their results. In this paper, we contribute empirical evidence that transfer learning for NMT can be …

Content truncated to 500 words