SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2689–2695 May 2-6, 2023 ©2023 Association for Computational Linguistics SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains Koustava Goswami1,2∗, Lukas Lange2, Jun Araki3, Heike Adel2 1 Adobe Research Bangalore, India 2 Bosch Center for Artificial Intelligence, Renningen, Germany 3 Bosch Research North America koustavag@adobe.com, {Lukas.Lange,Heike.Adel}@de.bosch.com, Jun.Araki@us.bosch.com Abstract Prompting pre-trained language models leads to promising results across natural language processing tasks but is less effective when applied in low-resource domains, due to the domain gap between the pre-training data and the downstream task. In this work, we bridge this gap with a novel and lightweight prompting methodology called SwitchPrompt for the adaptation of language models trained on datasets from the general domain to diverse low-resource domains. Using domain-specific keywords with a trainable gated prompt, Switch- Prompt offers domain-oriented prompting, that is, effective guidance on the target domains for general-domain language models. Our few- shot experiments on three text classification benchmarks demonstrate the efficacy of the general-domain pre-trained language models when used with SwitchPrompt. They often even outperform their domain-specific counterparts trained with baseline state-of-the-art prompt- ing methods by up to 10.7% performance in- crease in accuracy. This result indicates that SwitchPrompt effectively reduces the need for domain-specific language model pre-training. 1 Introduction Recent work showed promising results on different natural language processing tasks when prompting pre-trained language models (LMs) instead of fine- tuning them, especially in low-resource settings (Schucher et al., 2022). Most LMs which are pub- licly available have been trained on general-domain corpora (Devlin et al., 2019; Liu et al., 2019; Goyal et al., 2021), such as Wikipedia or the BooksCor- pus (Zhu et al., 2015). Applying them to tasks from a special domain results in a domain gap. For some special domains, domain-specific LMs exists, e.g., Clinical BERT (Alsentzer et al., ∗Research work conducted during internship at Bosch Center for Artificial Intelligence. Contact: kous- tavag@adobe.com 2019) or BioBERT (Lee et al., 2019). However, pre-training deep language models requires large amounts of text data.1 While we can assume the availability of large-scale text data in the general domain, this assumption might not hold for low- resource domains, making the creation of domain- specific LMs challenging. Moreover, training dif- ferent models for each and every new domain might be inefficient from a computation point of view.2 Even if there are domain-specific texts and compu- tational resources available, domain-specific LMs may not be able to get sufficient domain-oriented guidance through traditional prompting techniques because, for instance, domain-specific knowledge might be represented by a large and diverse vocab- ulary. As a result, both prompting LMs from the general domain and from a special domain might be ineffective, especially in low-resource settings. Motivated by these challenges, we explore domain-oriented prompts and propose a novel and lightweight method, SwitchPrompt, to effec- tively retrieve domain-specific knowledge from pre-trained LMs. It extends the sequence of soft- prompting vectors with a sequence of vectors rep- resenting domain-specific keywords …

Content truncated to 500 words