Improving Zero-shot Translation with Language-Independent Constraints

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 13–23 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 13 Improving Zero-shot Translation with Language-Independent Constraints Ngoc-Quan Pham∗ and Jan Niehues+ and Thanh-Le Ha+ and Alex Waibel+ ∗ Karlsruhe Institute of Technology {ngoc.pham, thanh-le.ha, alex.waibel}@kit.edu + Maastricht University jan.niehues@maastrichtuniversity.nl Abstract An important concern in training multilin- gual neural machine translation (NMT) is to translate between language pairs unseen dur- ing training, i.e zero-shot translation. Improv- ing this ability kills two birds with one stone by providing an alternative to pivot translation which also allows us to better understand how the model captures information between lan- guages. In this work, we carried out an investigation on this capability of the multilingual NMT mod- els. First, we intentionally create an encoder architecture which is independent with respect to the source language. Such experiments shed light on the ability of NMT encoders to learn multilingual representations, in general. Based on such proof of concept, we were able to de- sign regularization methods into the standard Transformer model, so that the whole archi- tecture becomes more robust in zero-shot con- ditions. We investigated the behaviour of such models on the standard IWSLT 2017 multilin- gual dataset. We achieved an average improve- ment of 2.23 BLEU points across 12 language pairs compared to the zero-shot performance of a state-of-the-art multilingual system. Ad- ditionally, we carry out further experiments in which the effect is confirmed even for lan- guage pairs with multiple intermediate pivots. 1 Introduction Neural machine translation (NMT) exploits neu- ral networks to directly learn to transform sen- tences from a source language to a target lan- guage (Sutskever et al., 2014; Bahdanau et al., 2014). Universal multilingual NMT discovered that a neural translation system can be trained on datasets containing source and target sentences in multiple languages (Firat et al., 2016; Johnson et al., 2016). Successfully trained models using this approach can be used to translate arbitrar- ily between any languages included in the train- ing data. In low-resource scenarios, multilingual NMT has proven to be an extremely useful reg- ularization method since each language direction benefits from the information of the others (Ha et al., 2016; Gu et al., 2018). An important research focus of multilingual NMT is zero-shot translation (ZS), or translation between languages included in multilingual data for which no directly parallel training data exists. Application-wise, ZS offers a faster and more di- rect path between languages compared to pivot translation, which requires translation to one or many intermediate languages. This can result in large latency and error propagation, common is- sues in non-end-to-end pipelines.From a represen- tation learning point of view, there is evidence of NMT’s ability to capture language-independent features, which have proved useful for cross- lingual transfer learning (Zoph et al., 2016; Kim et al., 2019) and provide motivation for ZS trans- lation. However it is still unclear if minimizing the difference in representations between languages is beneficial for zero-shot learning. …

Content truncated to 500 words