Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 528–537 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 528 Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation Annette Rios1 and Mathias M¨uller1 and Rico Sennrich1,2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh Abstract Zero-shot neural machine translation is an at- tractive goal because of the high cost of ob- taining data and building translation systems for new translation directions. However, pre- vious papers have reported mixed success in zero-shot translation. It is hard to predict in which settings it will be effective, and what limits performance compared to a fully su- pervised system. In this paper, we investi- gate zero-shot performance of a multilingual EN↔{FR,CS,DE,FI} system trained on WMT data. We find that zero-shot performance is highly unstable and can vary by more than 6 BLEU between training runs, making it diffi- cult to reliably track improvements. We ob- serve a bias towards copying the source in zero-shot translation, and investigate how the choice of subword segmentation affects this bias. We find that language-specific subword segmentation results in less subword copying at training time, and leads to better zero-shot performance compared to jointly trained seg- mentation. A recent trend in multilingual mod- els is to not train on parallel data between all language pairs, but have a single bridge lan- guage, e.g. English. We find that this nega- tively affects zero-shot translation and leads to a failure mode where the model ignores the language tag and instead produces English output in zero-shot directions. We show that this bias towards English can be effectively re- duced with even a small amount of parallel data in some of the non-English pairs. 1 Introduction Zero-shot translation has first been introduced by Firat et al. (2016) and refers to the ability of a mul- tilingual NMT model to translate between all its source and target languages, even those pairs for which no parallel data was seen in training. In the simplest setting, all parameters in the network are shared between the different languages and the translation is guided only by special tags to indi- cate the desired output language (Johnson et al., 2017; Ha et al., 2016). While this capability is attractive because it is an alternative to building N2 dedicated translation systems to serve N lan- guages, performance on zero-shot pairs tends to lag behind pivot translation. Recent papers, such as Arivazhagan et al. (2019), Gu et al. (2019) and Zhang et al. (2020), have suggested training tech- niques to improve the generalization to unseen lan- guage pairs, but performance varies considerably across settings. In this paper, we examine in detail the behavior of the multilingual model proposed by Johnson et al. (2017) on zero-shot translation directions. Our experiments show the following: • Translation quality for zero-shot language pairs is highly unstable between different training runs, and between training check- points, which calls for more rigour to avoid false positive results. …

Content truncated to 500 words