Efficient Domain Adaptation of Sentence Embeddings Using Adapters

Proceedings of Recent Advances in Natural Language Processing, pages 1046–1053 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_112 1046 Efficient Domain Adaptation of Sentence Embeddings Using Adapters Tim Schopf, Dennis N. Schneider, and Florian Matthes Technical University of Munich, Department of Computer Science, Germany {tim.schopf,dennis.schneider,matthes}@tum.de Abstract Sentence embeddings enable us to capture the semantic similarity of short texts. Most sen- tence embedding models are trained for general semantic textual similarity tasks. Therefore, to use sentence embeddings in a particular do- main, the model must be adapted to it in order to achieve good results. Usually, this is done by fine-tuning the entire sentence embedding model for the domain of interest. While this approach yields state-of-the-art results, all of the model’s weights are updated during fine- tuning, making this method resource-intensive. Therefore, instead of fine-tuning entire sen- tence embedding models for each target domain individually, we propose to train lightweight adapters. These domain-specific adapters do not require fine-tuning all underlying sentence embedding model parameters. Instead, we only train a small number of additional parameters while keeping the weights of the underlying sentence embedding model fixed. Training domain-specific adapters allows always using the same base model and only exchanging the domain-specific adapters to adapt sentence em- beddings to a specific domain. We show that using adapters for parameter-efficient domain adaptation of sentence embeddings yields com- petitive performance within 1% of a domain- adapted, entirely fine-tuned sentence embed- ding model while only training approximately 3.6% of the parameters. 1 Introduction Learning sentence embeddings is an essential task in natural language processing (NLP) and has al- ready been extensively investigated in the literature (Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Logeswaran and Lee, 2018; Cer et al., 2018; Reimers and Gurevych, 2019; Gao et al., 2021; Wu et al., 2022; Schopf et al., 2023d,a). Sentence embeddings are especially useful in information retrieval (Lewis et al., 2020; Schopf et al., 2022; Sentence Embedding Model Sentence General Embedding Sentence Embedding Model Sentence Domain-specific Embedding + Domain- specific Adapter Figure 1: Sentence embedding models are usually trained to obtain state-of-the-art sentence representa- tions for general semantic textual similarity tasks. By injecting domain-specific knowledge of adapters into the sentence embedding model, we can efficiently adapt the resulting representations for semantic textual simi- larity tasks in different domains. Schneider et al., 2022) or unsupervised text clas- sification settings (Schopf et al., 2021, 2023b,c). Lately, the most popular approach for sentence em- bedding learning is to fine-tune pretrained language models with a contrastive learning objective (Liu et al., 2021; Zhang et al., 2022; Chuang et al., 2022; Nishikawa et al., 2022; Cao et al., 2022; Jiang et al., 2022). While this approach provides state-of-the- art results, all of the model’s weights are updated during fine-tuning, making this method resource- intensive. This is a problem, particularly when domain-specific models are needed. Then, a spe- cialized model must be trained for each domain of interest, resulting in resource-intensive training. Recently, adapters have emerged as a parameter- efficient strategy to fine-tune …

Content truncated to 500 words