AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 2054–2063 May 2-6, 2023 ©2023 Association for Computational Linguistics AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models Alexandra Chronopoulou⋆†▽ Matthew E. Peters‡ Alexander Fraser▽ Jesse Dodge⋆‡ ▽Center for Information and Language Processing, LMU Munich, Germany ▽Munich Center for Machine Learning, Germany ‡Allen Institute for Artificial Intelligence, Seattle, WA Abstract Pretrained language models (PLMs) are trained on massive corpora, but often need to special- ize to specific domains. A parameter-efficient adaptation method suggests training an adapter for each domain on the task of language mod- eling. This leads to good in-domain scores but can be impractical for domain- or resource- restricted settings. A solution is to use a related- domain adapter for the novel domain at test time. In this paper, we introduce Adapter- Soup, an approach that performs weight-space averaging of adapters trained on different do- mains. Our approach is embarrassingly par- allel: first, we train a set of domain-specific adapters; then, for each novel domain, we deter- mine which adapters should be averaged at test time. We present extensive experiments show- ing that AdapterSoup consistently improves performance to new domains without extra training. We also explore weight averaging of adapters trained on the same domain with different hyper-parameters, and show that it preserves the performance of a PLM on new do- mains while obtaining strong in-domain results. We explore various approaches for choosing which adapters to combine, such as text cluster- ing and semantic similarity. We find that using clustering leads to the most competitive results on novel domains. 1 Introduction Large LMs are pre-trained using massive amounts of data in a self-supervised way (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019) and obtain general-domain knowledge. In or- der to adapt them to a new domain, continuing train- ing using in-domain data has been shown to be help- ful (Han and Eisenstein, 2019; Lee et al., 2020; Gu- rurangan et al., 2020). To avoid fine-tuning all pa- rameters, efficient methods such as domain-specific mixtures-of-experts (Gururangan et al., 2022) and ⋆Correspondence to achron@cis.lmu.de or jessed@allenai.org †Work done during an internship at Allen AI Select adapters for new domain Train an adapter per domain D1, D2, . . . , Dk Domain Adapter Domain Adapter PLM Domain Adapter θm . . . 1 l (θD1 + θD2 + θD3 + . . . + θDk−1) Domain Adapter Adapter Soup Weight-average selected adapters θD1 Domain Adapter Domain Adapter Domain Adapter θD3 . . . 1 n(θD1 + θD4 + . . . + θDk) Domain Adapter Adapter Soup θD1 θD2 θDk X PLM θm X X θD2 θD3 θDk Domain Adapter θD1 Domain Adapter θD2 Domain Adapter θD3 Domain Adapter θDk Figure 1: Illustration of AdapterSoup. Starting from the same random seed, an adapter is trained for each domain (domain adapter) on top of a PLM. Adapter- Soup averages the weights of the adapters that are most related to the new domain …

Content truncated to 500 words