Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

Findings of the Association for Computational Linguistics: ACL 2022, pages 582 - 588 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training Thuy-Trang Vu♢∗ Shahram Khadivi† Dinh Phung♢ Gholamreza Haffari♢ ♢Department of Data Science and AI, Monash University, Australia † eBay Inc. {trang.vuthithuy,first.last}@monash.edu skhadivi@ebay.com Abstract Generalising to unseen domains is under- explored and remains a challenge in neural ma- chine translation. Inspired by recent research in parameter-efficient transfer learning from pre- trained models, this paper proposes a fusion- based generalisation method that learns to com- bine domain-specific parameters. We propose a leave-one-domain-out training strategy to avoid information leaking to address the challenge of not knowing the test domain during training time. Empirical results on three language pairs show that our proposed fusion method outper- forms other baselines up to +0.8 BLEU score on average. 1 Introduction Building robust machine translation (MT) mod- els that can perform well on a test set outside the domain of training examples is highly desired in real-world scenarios. Despite recent great progress in neural machine translation (NMT) research, NMT models have been found sensitive to distri- bution shift and adversarial examples (Koehn and Knowles, 2017; Belinkov and Bisk, 2018; Müller et al., 2020). While improving an NMT model to a new domain has been studied extensively in domain adaptation settings where in-domain par- allel or monolingual data is given (Chu and Wang, 2018), generalising NMT models to unseen do- mains is under-explored (Specia et al., 2020). Domain generalisation is a problem setting in machine learning that tackles the challenge of learn- ing a robust model for unseen domains from mul- tiple existing domains. This problem is closely related to several settings such as multi-task learn- ing, transfer learning, and domain adaptation in terms of learning models from one or more given tasks/domains to enhance performance on some target tasks/domains. The main difference and chal- lenge in domain generalisation is that the test do- ∗Work done while doing internship at eBay Inc. mains are unknown in advance. Previous works on domain generalisation focused on learning in- variant features by minimising the difference in the representations of the given domains for the classification tasks (Li et al., 2018; Wang et al., 2020b; Gulrajani and Lopez-Paz, 2020). Learning a domain-invariant representation is applicable to the classification problems where such invariances may be sufficient to predict the target classes. How- ever, it may be inadequate for translation tasks. A good translation should not only preserve the invari- ant features such as syntax and grammar, but also be able to maintain the domain-specific features such as style of the source sentence. In this paper, we propose a fusion-based ap- proach to the domain generalisation problem for NMT. Our method comprises two training stages. The first stage is to learn domain-specific fea- tures through adapter modules added to the pre- trained encoder-decoder model. Previous works have shown that the task-specific adapter is an ef- fective alternative method to fine-tuning. It allows fast adaptation …

Content truncated to 500 words