Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation

Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation Aizhan Imankulova† Raj Dabre‡ Atsushi Fujita‡ Kenji Imamura‡ †Tokyo Metropolitan University 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan imankulova-aizhan@ed.tmu.ac.jp ‡National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {raj.dabre, atsushi.fujita, kenji.imamura}@nict.go.jp Abstract This paper proposes a novel multilin- gual multistage fine-tuning approach for low-resource neural machine translation (NMT), taking a challenging Japanese– Russian pair for benchmarking. Al- though there are many solutions for low- resource scenarios, such as multilingual NMT and back-translation, we have em- pirically confirmed their limited success when restricted to in-domain data. We therefore propose to exploit out-of-domain data through transfer learning, by us- ing it to first train a multilingual NMT model followed by multistage fine-tuning on in-domain parallel and back-translated pseudo-parallel data. Our approach, which combines domain adaptation, multilin- gualism, and back-translation, helps im- prove the translation quality by more than 3.7 BLEU points, over a strong baseline, for this extremely low-resource scenario. 1 Introduction Neural machine translation (NMT) (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015) has enabled end-to-end training of a trans- lation system without needing to deal with word alignments, translation rules, and complicated de- coding algorithms, which are the characteristics of phrase-based statistical machine translation (PB- SMT) (Koehn et al., 2007). Although NMT can be significantly better than PBSMT in resource- rich scenarios, PBSMT performs better in low- resource scenarios (Koehn and Knowles, 2017). c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. Only by exploiting cross-lingual transfer learn- ing techniques (Firat et al., 2016; Zoph et al., 2016; Kocmi and Bojar, 2018), can the NMT per- formance approach PBSMT performance in low- resource scenarios. However, such methods usually require an NMT model trained on a resource-rich language pair like French↔English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek↔English (child). On the other hand, multi- lingual approaches (Johnson et al., 2017) propose to train a single model to translate multiple lan- guage pairs. However, these approaches are effec- tive only when the parent target or source language is relatively resource-rich like English (En). Fur- thermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of do- main adaptation (Chu et al., 2017). In this paper, we work on a linguisti- cally distant and thus challenging language pair Japanese↔Russian (Ja↔Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor. Furthermore, the amount of indirect in-domain parallel corpora, i.e., Ja↔En and Ru↔En, are also small. As we demonstrate in Section 4, this severely limits the performance of prominent low-resource techniques, such as mul- tilingual modeling, back-translation, and pivot- based PBSMT. To remedy this, we propose a novel multistage fine-tuning method for NMT that com- bines multilingual modeling (Johnson et al., 2017) and domain adaptation (Chu et al., 2017). …

Content truncated to 500 words