Improving Zero-Shot Translation by Disentangling Positional Information

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1259–1273 August 1–6, 2021. ©2021 Association for Computational Linguistics 1259 Improving Zero-Shot Translation by Disentangling Positional Information Danni Liu1, Jan Niehues1, James Cross2, Francisco Guzm´an2, Xian Li2 1Department of Data Science and Knowledge Engineering, Maastricht University 2Facebook AI {danni.liu,jan.niehues}@maastrichtuniversity.nl {jcross,fguzman,xianl}@fb.com Abstract Multilingual neural machine translation has shown the capability of directly translating be- tween language pairs unseen in training, i.e. zero-shot translation. Despite being conceptu- ally attractive, it often suffers from low output quality. The difficulty of generalizing to new translation directions suggests the model repre- sentations are highly specific to those language pairs seen in training. We demonstrate that a main factor causing the language-specific rep- resentations is the positional correspondence to input tokens. We show that this can be easily alleviated by removing residual connec- tions in an encoder layer. With this mod- ification, we gain up to 18.5 BLEU points on zero-shot translation while retaining qual- ity on supervised directions. The improve- ments are particularly prominent between re- lated languages, where our proposed model outperforms pivot-based translation. More- over, our approach allows easy integration of new languages, which substantially expands translation coverage. By thorough inspections of the hidden layer outputs, we show that our approach indeed leads to more language- independent representations.1 1 Introduction Multilingual neural machine translation (NMT) sys- tem encapsulates several translation directions in a single model (Firat et al., 2017; Johnson et al., 2017). These multilingual models have been shown to be capable of directly translating between lan- guage pairs unseen in training (Johnson et al., 2017; Ha et al., 2016). Zero-shot translation as such is attractive both practically and theoretically. Com- pared to pivoting via an intermediate language, the direct translation halves inference-time computa- 1Code and scripts available in: https://github. com/nlp-dke/NMTGMinor/tree/master/ recipes/zero-shot encoder a big cat encoder un gato grande h1 h2 h3 h1 h3 h2 Figure 1: An example of language-specific encoder outputs as a results of the strong positional correspon- dence to input tokens (even assuming the word embed- dings are cross-lingually mapped). tion and circumvents error propagation. Consider- ing data collection, zero-shot translation does not require parallel data for a potentially quadratic num- ber of language pairs, which is sometimes imprac- tical to acquire especially between low-resource languages. Using less supervised data in turn re- duces training time. From a modeling perspective, zero-shot translation calls for language-agnostic representations, which are likely more robust and can benefit low-resource translation directions. Despite the potential benefits, achieving high- quality zero-shot translation is a challenging task. Prior works (Arivazhagan et al., 2019; Zhang et al., 2020a; Rios et al., 2020) have shown that standard systems tend to generate poor outputs, sometimes in an incorrect target language. It has been further shown that the encoder-decoder model captures spurious correlations between language pairs with supervised data (Gu et al., 2019). During training, the model only learns to encode the inputs in a form …

Content truncated to 500 words