A Study of Residual Adapters for Multi-Domain Neural Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 617–628 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 617 A Study of Residual Adapters for Multi-Domain Neural Machine Translation MinhQuang Pham†‡, Josep Crego†, Franc¸ois Yvon‡, Jean Senellart† †SYSTRAN / 5 rue Feydeau, 75002 Paris, France firstname.lastname@systrangroup.com ‡Universit´e Paris-Saclay, CNRS, LIMSI, 91405 Orsay, France firstname.lastname@limsi.fr Abstract Domain adaptation is an old and vexing prob- lem for machine translation systems. The most common and successful approach to su- pervised adaptation is to fine-tune a baseline system with in-domain parallel data. Stan- dard fine-tuning however modifies all the net- work parameters, which makes this approach computationally costly and prone to overfit- ting. A recent, lightweight approach, instead augments a baseline model with supplemen- tary (small) adapter layers, keeping the rest of the model unchanged. This has the addi- tional merit to leave the baseline model in- tact and adaptable to multiple domains. In this paper, we conduct a thorough analysis of the adapter model in the context of a multido- main machine translation task. We contrast multiple implementations of this idea using two language pairs. Our main conclusions are that residual adapters provide a fast and cheap method for supervised multi-domain adapta- tion; our two variants prove as effective as the original adapter model and open perspective to also make adapted models more robust to label domain errors. 1 Introduction Owing to multiple improvements, Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) nowadays delivers useful out- puts for many language pairs. However, as many deep learning models, NMT systems need to be trained with sufficiently large amounts of data to reach their best performance. Therefore, the quality of the translation of NMT models is still limited in low-resource language or domain conditions (Duh et al., 2013; Zoph et al., 2016; Koehn and Knowles, 2017). While many approaches have been pro- posed to improve the quality of NMT models in low-resource domains (see the recent survey of Chu and Wang (2018)), full fine-tuning (Luong and Manning, 2015; Neubig and Hu, 2018) of a generic baseline model remains the dominant supervised approach when adapting NMT models to specific domains. Under this view, building adapted systems is a two-step process: (a) one first trains NMT with the largest possible parallel corpora, aggregating texts from multiple, heterogeneous sources; (b) assum- ing that in-domain parallel documents are avail- able for the domain of interest, one then adapts the pre-trained model by resuming training with the sole in-domain corpus. It is a conjecture that the pretrained model constitutes a better initialization than a random one, especially when adaptation data is scarce. Indeed, studies of transfer learning for NMT such as Artetxe et al. (2020); Aji et al. (2020) have confirmed this claim in extensive experiments. Full fine-tuning, that adapts all the parameters of a baseline model usually significantly improves the quality of the NMT for the chosen domain. How- ever, it also yields large losses …

Content truncated to 500 words