Prompt-based Distribution Alignment for Domain Generalization in Text Classification

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10147 - 10157 December 7-11, 2022 ©2022 Association for Computational Linguistics Prompt-based Distribution Alignment for Domain Generalization in Text Classification Chen Jia†‡ and Yue Zhang‡§ †Fudan University, China ‡School of Engineering, Westlake University, China §Institute of Advanced Technology, Westlake Institute for Advanced Study, China {jiachen,zhangyue}@westlake.edu.cn Abstract Prompt-based learning (a.k.a. prompting) achieves high performance by bridging the gap between the objectives of language modeling and downstream tasks. Domain generalization ability can be improved by prompting since classification across different domains can be unified into the prediction of the same set of label words. The remaining challenge for do- main generalization by prompting comes from discrepancies between the data distribution of different domains. To improve domain general- ization with prompting, we learn distributional invariance across source domains via two align- ment regularization loss functions. The first is vocabulary distribution alignment, which uses a Kullback-Leibler divergence regularization on source-domain vocabulary distributions. The second is feature distribution alignment, which uses a novel adversarial training strategy to learn domain invariant representation across source domains. Experiments on sentiment analysis and natural language inference show the effectiveness of our method and achieve state-of-the-art results on six datasets. 1 Introduction Pretrained language models (PLMs) have achieved promising results on a range of natural language processing (NLP) tasks (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020). The framework of tuning a PLM by task data has achieved competi- tive performance (Devlin et al., 2019). However, it suffers significant performance degradation when the tuned PLM is directly applied to out-of-domain examples (Gururangan et al., 2020). To tackle the problem of domain shift where the training set and the test set come from different data distributions, unsupervised domain adaptation (Pan and Yang, 2009; Mansour et al., 2009) uses unlabeled target data cooperated with the labeled source data for training. However, in many real-world systems, access to unlabeled data in the target domain is also impossible. This paper focuses on domain generalization (DG), the practical and challenge setting, where a model trained on multiple source domains can be directly generalized to a target do- main without any labeled or unlabeled data from the target domain (Blanchard et al., 2011; Muandet et al., 2013). Prompt-based learning (a.k.a. Prompting) (Brown et al., 2020; Gao et al., 2021; Han et al., 2021a; Liu et al., 2021a) makes better use of pre- trained knowledge by bridging the gap between objectives of language modeling and downstream task. In particular, prompt-based text classification uses an identical projection from the probability distribution on label worlds to the probability dis- tribution on classification classes and has achieved state-of-the-art results (Gao et al., 2021; Han et al., 2021b; Hu et al., 2022). Intuitively, DG benefits from prompting in that different domains can share a unified set of label words, and thus no additional parameters such as output layers can carry domain-specific informa- tion to hinder the generalization performance for an unseen domain. However, there …

Content truncated to 500 words