Domain Confused Contrastive Learning for Unsupervised Domain Adaptation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2982 - 2995 July 10-15, 2022 ©2022 Association for Computational Linguistics Domain Confused Contrastive Learning for Unsupervised Domain Adaptation Quanyu Long, Tianze Luo, Wenya Wang and Sinno Jialin Pan Nanyang Technological University, Singapore {quanyu001, tianze001, wangwy, sinnopan}@ntu.edu.sg Abstract In this work, we study Unsupervised Do- main Adaptation (UDA) in a challenging self- supervised approach. One of the difficulties is how to learn task discrimination in the ab- sence of target labels. Unlike previous litera- ture which directly aligns cross-domain distri- butions or leverages reverse gradient, we pro- pose Domain Confused Contrastive Learning (DCCL) to bridge the source and the target do- mains via domain puzzles, and retain discrim- inative representations after adaptation. Tech- nically, DCCL searches for a most domain- challenging direction and exquisitely crafts do- main confused augmentations as positive pairs, then it contrastively encourages the model to pull representations towards the other domain, thus learning more stable and effective domain invariances. We also investigate whether con- trastive learning necessarily helps with UDA when performing other data augmentations. Ex- tensive experiments demonstrate that DCCL significantly outperforms baselines. 1 Introduction Pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019) have yielded considerable improvements with datasets drawing from various sources. However, the lack of porta- bility of language model to adapt to a new textual domain remains a central issue (Gururangan et al., 2020), especially when the training set and testing set do not follow the same underlying distribution (changing of topic and genres) - usually referred to as domain shift. In this paper, we focus on studying Unsupervised Domain Adaptation (UDA). UDA aims at designing adaptation algorithms that at- tempt to generalize well on the target domain by learning from both labeled samples from the source domain and unlabeled samples from the target do- main. Studying UDA fits real-world scenarios since labeled data in the target domain is usually absent. Moreover, advances in UDA will also help out-of- Source Domain Target Domain Pre-trained LM Domain Domain Puzzles Pulling Source Domain Puzzles Target Figure 1: Domain puzzles that are domain-confused and overlook domain-related information, could be re- garded as lying in an intermediate domain that aims to pull source and target samples closer to each other and bridge the two domains by learning domain invariant representations. distribution generalizations (Ramponi and Plank, 2020; Krueger et al., 2021). Extensive algorithms have been proposed to miti- gate the domain shift problem, for example, domain adversarial neural network (DANN) (Ganin et al., 2016) and distribution matching (Zhuang et al., 2015). For DANN, the training process of joint optimization is unstable, requiring extensive effort to tune the hyperparameters (Shah et al., 2018; Du et al., 2020; Karouzos et al., 2021). As for distri- bution matching, it is very difficult to preserve the discriminative power of the model on the target task while trying to perform instance level alignment (Saito et …

Content truncated to 500 words