Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5113–5129 July 9-14, 2023 ©2023 Association for Computational Linguistics Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories Shizhe Diao♡˚, Tianyang Xu♠˚, Ruijia Xu♡, Jiawei Wang♣, Tong Zhang♡ ♡The Hong Kong University of Science and Technology {sdiaoaa, rxuaq, tongzhang}@ust.hk ♠Wuhan University tianyangxu@whu.edu.cn ♣Shanghai Jiao Tong University wjw_sjt@sjtu.edu.cn Abstract Pre-trained language models (PLMs) demon- strate excellent abilities to understand texts in the generic domain while struggling in a spe- cific domain. Although continued pre-training on a large domain-specific corpus is effective, it is costly to tune all the parameters on the domain. In this paper, we investigate whether we can adapt PLMs both effectively and effi- ciently by only tuning a few parameters. Specif- ically, we decouple the feed-forward networks (FFNs) of the Transformer architecture into two parts: the original pre-trained FFNs to main- tain the old-domain knowledge and our novel domain-specific adapters to inject domain- specific knowledge in parallel. Then we adopt a mixture-of-adapters gate to fuse the knowledge from different domain adapters dynamically. Our proposed Mixture-of-Domain-Adapters (MixDA) employs a two-stage adapter-tuning strategy that leverages both unlabeled data and labeled data to help the domain adap- tation: i) domain-specific adapter on unla- beled data; followed by ii) the task-specific adapter on labeled data. MixDA can be seam- lessly plugged into the pretraining-finetuning paradigm and our experiments demonstrate that MixDA achieves superior performance on in-domain tasks (GLUE), out-of-domain tasks (ChemProt, RCT, IMDB, Amazon), and knowledge-intensive tasks (KILT). Further analyses demonstrate the reliability, scalabil- ity, and efficiency of our method.1 1 Introduction Pre-trained language models (PLMs) have achieved a multitude of successful applications in natural language understanding (Devlin et al., 2018; Liu et al., 2019; He et al., 2021b) and generation (Lewis et al., 2019; Zhang et al., 2020; Yang et al., 2020; Brown et al., 2020). The predominant methodol- ogy for domain adaptation is fine-tuning on labeled *Equal Contribution. 1The code is available at https://github.com/ Amano-Aki/Mixture-of-Domain-Adapters. domain-specific data or continued pre-training (Gu- rurangan et al., 2020) on unlabeled domain-specific data. Although effective, both fine-tuning and con- tinued pre-training methods require tuning all the parameters of a PLM, raising high costs beyond many institutions’ reach. To mitigate this, multi- ple parameter-efficient fine-tuning (PEFT) methods are proposed, including prompt-based tuning (Gao et al., 2021; Liu et al., 2021b; Schick and Schütze, 2021; Li and Liang, 2021; Liu et al., 2021a), and adapter-based tuning (Houlsby et al., 2019; Pfeif- fer et al., 2020b; Hu et al., 2021). However, they are more concerned about task adaptation and it is still unclear how to regularly, and inexpensively inject domain knowledge into PLMs for different domain-specific tasks. Moreover, directly tuning PLMs on a domain-specific corpus with PEFT methods will lead to the catastrophic forgetting problem (Yogatama et al., 2019; Gururangan et al., 2020). These limitations highlight an important re- search question: how to adapt PLMs with the new domain knowledge while keeping the old-domain knowledge unmodified? Inspired by …

Content truncated to 500 words