Monolingual Adapters for Zero-Shot Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4465–4470, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4465 Monolingual Adapters for Zero-Shot Neural Machine Translation Jerin Philip†∗and Alexandre Bérard† and Matthias Gallé† and Laurent Besacier†‡ †NAVER LABS Europe ‡Université Grenoble Alpes jerin.philip@research.iiit.ac.in alexandre.berard@naverlabs.com laurent.besacier@univ-grenoble-alpes.fr matthias.galle@naverlabs.com Abstract We propose a novel adapter layer formalism for adapting multilingual models. They are more parameter-efficient than existing adapter layers while obtaining as good or better per- formance. The layers are specific to one lan- guage (as opposed to bilingual adapters) allow- ing to compose them and generalize to unseen language-pairs. In this zero-shot setting, they obtain a median improvement of +2.77 BLEU points over a strong 20-language multilingual Transformer baseline trained on TED talks. 1 Introduction Of the many virtues of multilingual neural machine translation (MNMT), arguably the most attractive is the promise of improving performance in the low resource setting (Johnson et al., 2017; Ari- vazhagan et al., 2019; Dabre et al., 2020). These models even allow for the extreme of these cases, namely to translate in language pair directions which are unseen at training time (zero-shot set- ting in this paper). Unfortunately, while perfor- mance in the low-resource setting indeed increases significantly, their zero-shot performance remains very low (Johnson et al., 2017). In this paper, we propose a neural architecture that allows to translate from any of the source languages towards any of the target languages seen in the training data, regardless of the presence of that specific language direction during training. For that, we build upon the recently proposed adapter layers for NMT (Bapna and Firat, 2019), by using mono- lingual (language-specific) adapter layers, instead of bilingual (language-pair specific) ones. This design difference improves their compositionality, permitting to combine any encoder adapter with other decoder adapters. Monolingual adapter lay- ers perform as good as bilingual adapter layers in ∗Work done during an internship at NAVER LABS Eu- rope. the non–zero-shot setting, are effective in the zero- shot setting and have the additional advantage of requiring fewer parameters. 2 Related Work Zero-shot translation is direct translation in a language pair unseen during training. Aharoni et al. (2019) analyze the zero-shot performance of MNMT models as a function of the number of language pairs. They observe that having more languages results in better zero-shot performance. However, several artifacts arise, as described by Dabre et al. (2020); Zhang et al. (2020); Aharoni et al. (2019); Arivazhagan et al. (2019), like off- target translation and insufficient modeling capac- ity of the MNMT models. Zhang et al. (2020) use language-aware layer normalization and lin- ear transformation to improve some drawbacks of MNMT; they also rely massively on back- translation to improve zero-shot translation. Adaptation to a new language pair may be addressed by training a multilingual model then fine-tuning it with parallel data in the language pair of interest (Neubig and Hu, 2018; Variš and Bojar, 2019; Stickland et al., 2020). Escolano et al. (2020) propose plug-and-play encoders and decoders per …

Content truncated to 500 words