Improving Zero-Shot Translation of Low-Resource Languages

Improving Zero-Shot Translation of Low-Resource Languages Surafel M. Lakew1,2, Quintino F. Lotito 2,⇤, Matteo Negri1, Marco Turchi1, Marcello Federico1 1Fondazione Bruno Kessler, Trento, Italy 2University of Trento, Trento, Italy federico@fbk.eu Abstract Recent work on multilingual neural machine translation re- ported competitive performance with respect to bilingual models and surprisingly good performance even on (zero- shot) translation directions not observed at training time. We investigate here a zero-shot translation in a particularly low- resource multilingual setting. We propose a simple iterative training procedure that leverages a duality of translations di- rectly generated by the system for the zero-shot directions. The translations produced by the system (sub-optimal since they contain mixed language from the shared vocabulary), are then used together with the original parallel data to feed and iteratively re-train the multilingual network. Over time, this allows the system to learn from its own generated and increasingly better output. Our approach shows to be effec- tive in improving the two zero-shot directions of our multi- lingual model. In particular, we observed gains of about 9 BLEU points over a baseline multilingual model and up to 2.08 BLEU over a pivoting mechanism using two bilingual models. Further analysis shows that there is also a slight im- provement in the non-zero-shot language directions. 1. Introduction Machine translation of low-resource languages represents a challenge for neural machine translation (NMT) [1]. Recent efforts in multilingual NMT (Multi-NMT) [2, 3] have shown to improve translation performance in low-resource settings. Multi-NMT models can be trained with parallel corpora of several language pairs to work in many-to-one, one-to-many, or many-to-many translation directions. A simple approach, named target-forcing [3], is to prepend to the source sentence a tag specifying the target language, both at training and test- ing time. In addition to performance gains for low-resource languages, the benefit of Multi-NMT is the possibility to per- form zero-shot translation, i.e. across directions that were not observed at training time. Application scenarios in which zero-shot translation can bootstrap the creation of new parallel data – e.g. via human post-editing– [2], show how translation performance in the initial zero-shot direction improves over time with the addi- tion of new parallel data. In this work, we explore instead the * Work done during a summer internship at FBK. Italian English Romanian German Dutch zero-shot pivoting pivoting Figure 1: Our zero-shot translation setting for Italian- Romanian. Parallel data is available only for Italian- English, Romanian-English, German-English, and Dutch- English. We leverage multi-lingual neural machine transla- tion trained on all available parallel data to translate across Italian-Romanian (in both directions), either directly (zero- shot) or through English (pivoting). possibility to enable a trained Multi-NMT model to further learn from its own generated data. Briefly, our method works as follows: first (1), we let the Multi-NMT engine generate zero-shot translations on some portion of the training data; then (2), we re-start the training process on both the gen- erated translations and the original parallel data. We repeat this training-inference-training cycle for a few times. Notice that, …

Content truncated to 500 words