Domain Generalization for Text Classification with Memory-Based Supervised Contrastive Learning

Proceedings of the 29th International Conference on Computational Linguistics, pages 6916–6926 October 12–17, 2022. 6916 Domain Generalization for Text Classification with Memory-Based Supervised Contrastive Learning Qingyu Tan∗1,2 Ruidan He†1 Lidong Bing1 Hwee Tou Ng2 1DAMO Academy, Alibaba Group 2Department of Computer Science, National University of Singapore {qingyu.tan,ruidan.he,l.bing}@alibaba-inc.com {qtan6,nght}@comp.nus.edu.sg Abstract While there is much research on cross-domain text classification, most existing approaches fo- cus on one-to-one or many-to-one domain adap- tation. In this paper, we tackle the more chal- lenging task of domain generalization, in which domain-invariant representations are learned from multiple source domains, without access to any data from the target domains, and clas- sification decisions are then made on test doc- uments in unseen target domains. We propose a novel framework based on supervised con- trastive learning with a memory-saving queue. In this way, we explicitly encourage examples of the same class to be closer and examples of different classes to be further apart in the em- bedding space. We have conducted extensive experiments on two Amazon review sentiment datasets, and one rumour detection dataset. Ex- perimental results show that our domain gen- eralization method consistently outperforms state-of-the-art domain adaptation methods1. 1 Introduction Text classification is a highly important and widely studied natural language understanding task. Re- cent success in self-supervised pre-training has significantly improved the state-of-the-art perfor- mance in sentiment analysis. However, senti- ment classification is widely known as a domain- dependent task, mainly because sentiment expres- sions can have different meanings in different do- mains (e.g., “long” in “long waiting time” of a restaurant review is negative, while in “long bat- tery life” of a laptop review is positive). More- over, the amount of labeled data is highly imbal- anced across different domains. Many NLP do- mains still lack sufficient labeled data for training ∗† Qingyu Tan is under the Joint PhD Program between Alibaba and National University of Singapore. † Corresponding author. 1Our code is available at https://github.com/ tonytan48/MSCL. a high-performance classifier. Therefore, it is cru- cial to adapt sentiment knowledge from resource- rich domains to low-resource domains. This strand of research is known as domain adaptation (DA). Prior works on domain adaptation typically follow a one-to-one (Jiang and Zhai, 2007) or many-to-one adaptation setting (Zhao et al., 2018), where the model is usually trained on labeled data from the source domain along with unlabeled data from the target domain and is then evaluated on the target do- main data. The usage of the unlabeled target data is crucial in DA. Prior works mainly use it for domain- invariant representation learning or model selection. More recently, Wright and Augenstein (2020) have demonstrated that large pretrained language models (PrLMs) are able to achieve promising performance for cross-domain sentiment classification. Subse- quent works further improve the adaptation per- formance through domain adversarial training (Du et al., 2020; Karouzos et al., 2021), iterative pseudo- labeling (Ye et al., 2020), or prompting (Ben-David et al., 2022), where they all adapt the DA setting and assume unlabeled target data is available …

Content truncated to 500 words