Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 578–598 November 10–11, 2021. ©2021 Association for Computational Linguistics 578 Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters Asa Cooper Stickland∗ University of Edinburgh a.cooper.stickland@ed.ac.uk Alexandre Bérard Vassilina Nikoulina NAVER LABS Europe first.last@naverlabs.com Abstract Adapter layers are lightweight, learnable units inserted between transformer layers. Recent work explores using such layers for neural ma- chine translation (NMT), to adapt pre-trained models to new domains or language pairs, training only a small set of parameters for each new setting (language pair or domain). In this work we study the compositionality of language and domain adapters in the con- text of Machine Translation. We aim to study, 1) parameter-efficient adaptation to multiple domains and languages simultaneously (full- resource scenario) and 2) cross-lingual transfer in domains where parallel data is unavailable for certain language pairs (partial-resource sce- nario). We find that in the partial resource sce- nario a naive combination of domain-specific and language-specific adapters often results in ‘catastrophic forgetting’ of the missing lan- guages. We study other ways to combine the adapters to alleviate this issue and maximize cross-lingual transfer. With our best adapter combinations, we obtain improvements of 3- 4 BLEU on average for source languages that do not have in-domain data. For target lan- guages without in-domain data, we achieve a similar improvement by combining adapters with back-translation. Supplementary material is available at https://tinyurl.com/ r66stbxj. 1 Introduction Multilingual Neural Machine Translation (NMT) has made a lot of progress recently (Johnson et al., 2017; Bapna and Firat, 2019; Aharoni et al., 2019; Zhang et al., 2020; Fan et al., 2020a) and is now widely adopted by the community and MT service providers. Multilingual NMT models handle multi- ple language directions at once and allow for knowl- edge transfer to low-resource languages. Machine ∗Work done during an internship at NAVER LABS Eu- rope. translation systems often need to be adapted to spe- cific domains like legal or medical text. However, when adapting multilingual systems, in-domain data for most language pairs might not exist. We would like to be able to leverage data in a subset of language pairs to transfer domain knowledge to other languages. Straightforward methods of domain adaptation include fine-tuning (Freitag and Al-Onaizan, 2016) or usage of domain tags (Kobus et al., 2017; Britz et al., 2017) for different domains. For these meth- ods each new domain request would require re- training the whole model, which is a costly proce- dure. And naive training on a subset of languages typically reduces performance on all other lan- guages (Garcia et al., 2021), a phenomenon known as ‘catastrophic forgetting’ (McCloskey and Cohen, 1989). An alternative technique for adapting such mod- els to new language-pairs or domains are ‘adapter layers’ (Bapna and Firat, 2019), lightweight, learn- able units inserted between transformer layers. A previously trained large multilingual model can be adapted to each language-pair by learning only these small units, and keeping the rest of the model frozen. This procedure …

Content truncated to 500 words