Zero-Resource Translation with Multi-Lingual Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Zero-Resource Translation with Multi-Lingual Neural Machine Translation Orhan Firat⋆ Middle East Technical University orhan.firat@ceng.metu.edu.tr Baskaran Sankaran IBM T.J. Watson Research Center Yaser Al-onaizan IBM T.J. Watson Research Center Fatos T. Yarman Vural Middle East Technical University Kyunghyun Cho New York University Abstract In this paper, we propose a novel finetuning algorithm for the recently introduced multi- way, multilingual neural machine translate that enables zero-resource machine transla- tion. When used together with novel many- to-one translation strategies, we empirically show that this finetuning algorithm allows the multi-way, multilingual model to translate a zero-resource language pair (1) as well as a single-pair neural translation model trained with up to 1M direct parallel sentences of the same language pair and (2) better than pivot- based translation strategy, while keeping only one additional copy of attention-related pa- rameters. 1 Introduction A recently introduced neural machine transla- tion (Forcada and ˜Neco, 1997; Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) has proven to be a platform for new opportu- nities in machine translation research. Rather than word-level translation with language-specific pre- processing, neural machine translation has found to work well with statistically segmented subword se- quences as well as sequences of characters (Chung et al., 2016; Luong and Manning, 2016; Sennrich et al., 2015b; Ling et al., 2015). Also, recent works show that neural machine translation provides a seamless way to incorporate multiple modalities ⋆ Work carried out while the author was at IBM Research. other than natural language text in translation (Lu- ong et al., 2015a; Caglayan et al., 2016). Further- more, neural machine translation has been found to translate between multiple languages, achieving better translation quality by exploiting positive lan- guage transfer (Dong et al., 2015; Firat et al., 2016; Zoph and Knight, 2016). In this paper, we conduct in-depth investiga- tion into the recently proposed multi-way, multilin- gual neural machine translation (Firat et al., 2016). Specifically, we are interested in its potential for zero-resource machine translation, in which there does not exist any direct parallel examples between a target language pair. Zero-resource translation has been addressed by pivot-based translation in tradi- tional machine translation research (Wu and Wang, 2007; Utiyama and Isahara, 2007; Habash and Hu, 2009), but we explore a way to use the multi-way, multilingual neural model to translate directly from a source to target language. In doing so, we begin by studying different trans- lation strategies available in the multi-way, multi- lingual model in Sec. 3–4. The strategies include a usual one-to-one translation as well as variants of many-to-one translation for multi-source transla- tion (Zoph and Knight, 2016). We empirically show that the many-to-one strategies significantly outper- form the one-to-one strategy. We move on to zero-resource translation by first evaluating a vanilla multi-way, multilingual model on a zero-resource language pair, which revealed that the vanilla model cannot do zero-resource trans- lation …

Content truncated to 500 words