Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1410–1416, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1410 Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation Raj Dabre Atsushi Fujita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan firstname.lastname@nict.go.jp Chenhui Chu Osaka University, Japan chu@ids.osaka-u.ac.jp Abstract This paper highlights the impressive utility of multi-parallel corpora for transfer learn- ing in a one-to-many low-resource neural ma- chine translation (NMT) setting. We report on a systematic comparison of multistage fine- tuning configurations, consisting of (1) pre- training on an external large (209k–440k) par- allel corpus for English and a helping target language, (2) mixed pre-training or fine-tuning on a mixture of the external and low-resource (18k) target parallel corpora, and (3) pure fine- tuning on the target parallel corpora. Our ex- periments confirm that multi-parallel corpora are extremely useful despite their scarcity and content-wise redundancy thus exhibiting the true power of multilingualism. Even when the helping target language is not one of the target languages of our concern, our multistage fine- tuning can give 3–9 BLEU score gains over a simple one-to-one model. 1 Introduction Encoder-decoder based neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015) allows for end-to-end training of a transla- tion model. While NMT is known to perform well for resource-rich language pairs, such as French– English and German–English (Bojar et al., 2018), it performs poorly for resource-poor pairs (Zoph et al., 2016; Riza et al., 2016). Translating from a resource-poor language into English is typically easier than the other direc- tion, because large parallel corpora between En- glish and other languages can be used for trans- fer learning (Zoph et al., 2016) and many-to-one modeling.1 Additionally, large English monolin- gual data can be back-translated to obtain pseudo- parallel corpora for augmenting the performance of such NMT models. 1Note that “many-to-one” does not mean multi-source machine translation (Zoph and Knight, 2016). Translating from English into a resource-poor language is substantially more difficult. Dong et al. (2015) have shown that a one-to-many model trained on middle-sized parallel data (200k sen- tence pairs) can improve the translation quality over a one-to-one model. However, it is unclear whether this works for much resource-poorer, more distant, and more diverse language pairs. Using pseudo-parallel data is a potential solution, but for most resource-poor languages, the amount of available clean and in-domain monolingual data are limited. It is also unclear what the real reason behind improvements in translation is: increase in the training data or multilingualism. This paper focuses on (a) improving the per- formance of NMT for translating English to low- resource languages (b) via exploiting multilingual- ism (c) through transfer learning based on multi- stage fine-tuning. The power of multilingualism is verified by using multi-parallel corpora, i.e., the same text in different languages.2 Unlike previ- ous studies that only apply single-stage fine-tuning for one-to-one …

Content truncated to 500 words