Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1258–1268 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1258 Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations Jiatao Gu* ♦, Yong Wang* †, Kyunghyun Cho ♦‡ and Victor O.K. Li † ♦Facebook AI Research †The University of Hong Kong ‡New York University, CIFAR Azrieli Global Scholar ♦{jgu, kyunghyuncho}@fb.com †{wangyong, vli}@eee.hku.hk Abstract Zero-shot translation, translating between lan- guage pairs on which a Neural Machine Trans- lation (NMT) system has never been trained, is an emergent property when training the system in multilingual settings. However, na¨ıve train- ing for zero-shot NMT easily fails, and is sen- sitive to hyper-parameter setting. The perfor- mance typically lags far behind the more con- ventional pivot-based approach which trans- lates twice using a third language as a pivot. In this work, we address the degeneracy prob- lem due to capturing spurious correlations by quantitatively analyzing the mutual informa- tion between language IDs of the source and decoded sentences. Inspired by this analysis, we propose to use two simple but effective ap- proaches: (1) decoder pre-training; (2) back- translation. These methods show significant improvement (4 ∼ 22 BLEU points) over the vanilla zero-shot translation on three challeng- ing multilingual datasets, and achieve similar or better results than the pivot-based approach. 1 Introduction Despite the recent domination of neural network- based models (Sutskever et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017) in the field of machine translation, which have fewer pipelined components and significantly outperform phrase- based systems (Koehn et al., 2003), Neural Ma- chine Translation (NMT) still works poorly when the available number of training examples is lim- ited. Research on low-resource languages is draw- ing increasing attention, and it has been found promising to train a multilingual NMT (Firat et al., 2016a) model for high- and row-resource lan- guages to deal with low-resource translation (Gu et al., 2018b). As an extreme in terms of the num- ber of supervised examples, prior works dug into * Equal contribution. translation with zero-resource (Firat et al., 2016b; Chen et al., 2017; Lample et al., 2018a,b) where the language pairs in interest do not have any par- allel corpora between them. In particular, John- son et al. (2017) observed an emergent property of zero-shot translation where a trained multilingual NMT model is able to automatically do translation on unseen language pairs; we refer to this setting as zero-shot NMT from here on. In this work, we start with a typical degen- eracy issue of zero-shot NMT, reported in sev- eral recent works (Arivazhagan et al., 2018; Ses- torain et al., 2018), that zero-shot NMT is sen- sitive to training conditions, and the translation quality usually lags behind the pivot-based meth- ods which use a shared language as a bridge for translation (Utiyama and Isahara, 2007; Cheng et al., 2016; Chen et al., 2017). We first quanti- tatively show that this degeneracy issue of zero- shot NMT is …

Content truncated to 500 words