Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 866–876, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 866 Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages Yunsu Kim1∗ Petre Petrov1,2∗ Pavel Petrushkov2 Shahram Khadivi2 Hermann Ney1 1RWTH Aachen University, Aachen, Germany {surname}@cs.rwth-aachen.de 2eBay, Inc., Aachen, Germany {petrpetrov,ppetrushkov,skhadivi}@ebay.com Abstract We present effective pre-training strategies for neural machine translation (NMT) using parallel corpora involving a pivot language, i.e., source-pivot and pivot-target, leading to a significant improvement in source→target translation. We propose three methods to increase the relation among source, pivot, and target languages in the pre-training: 1) step-wise training of a single model for different language pairs, 2) additional adapter component to smoothly connect pre-trained encoder and decoder, and 3) cross-lingual encoder training via autoencoding of the pivot language. Our methods greatly out- perform multilingual models up to +2.6% BLEU in WMT 2019 French→German and German→Czech tasks. We show that our improvements are valid also in zero-shot/zero- resource scenarios. 1 Introduction Machine translation (MT) research is biased to- wards language pairs including English due to the ease of collecting parallel corpora. Trans- lation between non-English languages, e.g., French→German, is usually done with pivot- ing through English, i.e., translating French (source) input to English (pivot) first with a French→English model which is later translated to German (target) with a English→German model (De Gispert and Marino, 2006; Utiyama and Isahara, 2007; Wu and Wang, 2007). How- ever, pivoting requires doubled decoding time and the translation errors are propagated or expanded via the two-step process. Therefore, it is more beneficial to build a sin- gle source→target model directly for both effi- ciency and adequacy. Since non-English language ∗ Equal contribution. pairs often have little or no parallel text, common choices to avoid pivoting in NMT are generating pivot-based synthetic data (Bertoldi et al., 2008; Chen et al., 2017) or training multilingual systems (Firat et al., 2016; Johnson et al., 2017). In this work, we present novel transfer learn- ing techniques to effectively train a single, di- rect NMT model for a non-English language pair. We pre-train NMT models for source→pivot and pivot→target, which are transferred to a source→target model. To optimize the usage of given source-pivot and pivot-target parallel data for the source→target direction, we devise the fol- lowing techniques to smooth the discrepancy be- tween the pre-trained and final models: • Step-wise pre-training with careful parameter freezing. • Additional adapter component to familiarize the pre-trained decoder with the outputs of the pre-trained encoder. • Cross-lingual encoder pre-training with au- toencoding of the pivot language. Our methods are evaluated in two non-English language pairs of WMT 2019 news translation tasks: high-resource (French→German) and low- resource (German→Czech). We show that NMT models pre-trained with our methods are highly ef- fective in various data conditions, when fine-tuned for source→target with: • Real parallel corpus • Pivot-based synthetic parallel corpus (zero- resource) • None (zero-shot) For each data …

Content truncated to 500 words