Multilingual Unsupervised Neural Machine Translation with Denoising Adapters

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6650–6662 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6650 Multilingual Unsupervised Neural Machine Translation with Denoising Adapters Ahmet Üstün†∗ Alexandre Bérard‡ Laurent Besacier‡ Matthias Gallé‡ †University of Groningen ‡NAVER LABS Europe a.ustun@rug.nl first.last@naverlabs.com Abstract We consider the problem of multilingual un- supervised machine translation, translating to and from languages that only have monolin- gual data by using auxiliary parallel language pairs. For this problem the standard proce- dure so far to leverage the monolingual data is back-translation, which is computationally costly and hard to tune. In this paper we propose instead to use denois- ing adapters, adapter layers with a denoising objective, on top of pre-trained mBART-50. In addition to the modularity and flexibility of such an approach we show that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally. 1 Introduction Two major trends have in the last years provided surprising and exciting new avenues in Neural Ma- chine Translation (NMT). First, Multilingual Neu- ral Machine Translation (Firat et al., 2016; Ha et al., 2016; Johnson et al., 2017; Aharoni et al., 2019) has achieved impressive results on large-scale mul- tilingual benchmarks with diverse sets of language pairs. It has the advantage of resulting in only one model to maintain, as well as benefiting from cross-lingual knowledge transfer. Second, Unsuper- vised Neural Machine Translation (UNMT) (Lam- ple et al., 2018; Artetxe et al., 2018) allows to train translation systems from monolingual data only. Training bilingual UNMT systems (Conneau and Lample, 2019; Artetxe et al., 2019) often assume high-quality in-domain monolingual data and is mostly limited to resource-rich languages. In ad- dition to the pretraining and the denoising auto- encoding, they require one or more expensive steps of back-translation (Sennrich et al., 2016) in order to create an artificial parallel training corpus. ∗Work done during an internship at NAVER LABS Eu- rope. zz19 zz2 xx18 xx2 xx1 en zz1 zz17 zz16 Figure 1: Overview of our multilingual unsupervised NMT setup where dashed lines indicate 17 unsuper- vised languages without parallel data (zzn) and full lines indicate 19 auxiliary languages with parallel data for training (xxn). Adapted from Garcia et al. (2021). Multilingual UNMT aims at combining these two trends. As depicted in Fig 1, some auxil- iary languages have access to parallel data paired with English (en ↔ xx1), while unsupervised lan- guages only have monolingual data (zz1). The goal of such an approach is to make use of the auxil- iary parallel data to learn the translation task and hopefully transfer this task knowledge to the unsu- pervised languages. The end model should be able to translate to/from English in both the auxiliary and unsupervised languages. This setting has only been addressed very re- cently (Sun et al., 2020; Liu et al., 2020; Wang et al., 2021; Garcia et al., 2021). However all cur- rent approaches rely on back-translation, either of- fline or online. This …

Content truncated to 500 words