UDAPTER - Efficient Domain Adaptation Using Adapters

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2249–2263 May 2-6, 2023 ©2023 Association for Computational Linguistics UDAPTER - Efficient Domain Adaptation Using Adapters Bhavitvya Malik∗α, Abhinav Ramesh Kashyap ∗βγ, Min-Yen Kanβ, Soujanya Poria DeCLaRe α The University of Edinburgh, Edinburgh β National University of Singapore, Singapore γ ASUS Intelligent Cloud Services (AICS), Singapore DeCLaReDeCLaRe Lab, Singapore University of Technology and Design, Singapore b.malik-1@sms.ed.ac.uk, abhinav_kashyap@asus.com, kanmy@comp.nus.edu.sg, sporia@sutd.edu.sg Abstract We propose two methods to make unsuper- vised domain adaptation (UDA) more param- eter efficient using adapters, small bottleneck layers interspersed with every layer of the large- scale pre-trained language model (PLM). The first method deconstructs UDA into a two-step process: first by adding a domain adapter to learn domain-invariant information and then by adding a task adapter that uses domain- invariant information to learn task represen- tations in the source domain. The second method jointly learns a supervised classifier while reducing the divergence measure. Com- pared to strong baselines, our simple meth- ods perform well in natural language inference (MNLI) and the cross-domain sentiment classifi- cation task. We even outperform unsupervised domain adaptation methods such as DANN (Ganin et al., 2016) and DSN (Bousmalis et al., 2016) in sentiment classification, and we are within 0.85% F1 for natural language infer- ence task, by fine-tuning only a fraction of the full model parameters. We release our code at https://github.com/declare-lab/domadapter. 1 Introduction Fine-tuning pretrained language models (PLM) is the predominant method for improving NLP tasks such as sentiment analysis, natural language in- ference, and other language understanding tasks (Wang et al., 2018). However, fine-tuning forces us to modify all the parameters of the model and store one copy of the model for one task. Given the large size of current PLMs, this can be expensive. Furthermore, fine-tuning needs large-scale data to be effective and is unstable when using different seeds (Han et al., 2021). A new approach to alleviate this is parameter- efficient fine-tuning – freezing the PLM parameters ∗The first two authors contributed equally. and fine-tuning only a small fraction of the param- eters. Fine-tuning with adapters (Houlsby et al., 2019) is one of these methods in which small ad- ditional layers are tuned within each PLM layer. Fine-tuning with adapters has many advantages: performance comparable to full fine-tuning (He et al., 2021a), and robustness to different seeds and adversarial examples (Han et al., 2021). Unsupervised domain adaptation (UDA) aims to adapt models to new domains and considers situa- tions where labeled data are available only in the source domain and unlabeled data are available in the target domain. UDA methods in general have two components: The first reduces the divergence between the source and target domains, and the second reduces the loss corresponding to a particu- lar task (Ramesh Kashyap et al., 2021a). However, they fine-tune a large number of parameters and are susceptible to catastrophic forgetting. Adapters (Houlsby et al., 2019) can help solve these prob- lems. However, the benefits of using …

Content truncated to 500 words