Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5423–5433 August 1–6, 2021. ©2021 Association for Computational Linguistics 5423 Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation Bo Zhang1, Xiaoming Zhang1,∗, Yun Liu2, Lei Cheng3, Zhoujun Li2 1 School of Cyber Science and Technology, Beihang University, China 2 State Key Laboratory of Software Development Environment, Beihang University, China 3 Shenzhen Research Institute of Big Data, Shenzhen, China {zhangnet,yolixs,gzliuyun,lizj}@buaa.edu.cn, leicheng@sribd.cn Abstract Unsupervised Domain Adaptation (UDA) aims to transfer the knowledge of source do- main to the unlabeled target domain. Exist- ing methods typically require to learn to adapt the target model by exploiting the source data and sharing the network architecture across domains. However, this pipeline makes the source data risky and is inflexible for deploy- ing the target model. This paper tackles a novel setting where only a trained source model is available and different network ar- chitectures can be adapted for target domain in terms of deployment environments. We propose a generic framework named Cross- domain Knowledge Distillation (CdKD) with- out needing any source data. CdKD matches the joint distributions between a trained source model and a set of target data during dis- tilling the knowledge from the source model to the target domain. As a type of impor- tant knowledge in the source domain, for the first time, the gradient information is ex- ploited to boost the transfer performance. Ex- periments on cross-domain text classification demonstrate that CdKD achieves superior per- formance, which verifies the effectiveness in this novel setting. 1 Introduction Annotating sufficient training data is usually an expensive and time-consuming work for diverse application domains. Unsupervised Domain Adap- tation (UDA) aims at solving this learning prob- lem in the unlabeled target domain by utilizing the abundant knowledge in an existing domain called source domain, even when these domains may have different distributions. This technique has moti- vated research on cross-domain text classification (Chen et al., 2019; Ye et al., 2020; Gururangan et al., 2020). One of the important knowledge in the source domain is the labels of samples. Cur- rent methods mainly leverage the labeled source ∗ Corresponding author. data and unlabeled target data to learn the domain- invariant features (Tzeng et al., 2014; Ganin and Lempitsky, 2015) and the discriminative features (Saito et al., 2017; Ge et al., 2020) that are shared across different domains. Unfortunately, sometimes we are forbidden ac- cess to the source data, which are distributed on different devices and usually contain private infor- mation, e.g., user profile. Existing methods cannot solve the UDA problem without the source data yet. In addition, it is necessary to adapt the target do- main with a flexible network architecture different from the source domain in terms of different de- ployment requirements for different domains. But most of works (Liang et al., 2020; Li et al., 2020) are required to share the same network architecture between different …

Content truncated to 500 words