A Comparison of Strategies for Source-Free Domain Adaptation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8352 - 8367 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Comparison of Strategies for Source-Free Domain Adaptation Xin Su Yiyun Zhao Steven Bethard University of Arizona Tucson, AZ, USA {xinsu, yiyunzhao, bethard}@email.arizona.edu Abstract Data sharing restrictions are common in NLP, especially in the clinical domain, but there is limited research on adapting models to new domains without access to the original train- ing data, a setting known as source-free do- main adaptation. We take algorithms that tra- ditionally assume access to the source-domain training data—active learning, self-training, and data augmentation—and adapt them for source-free domain adaptation. Then we sys- tematically compare these different strategies across multiple tasks and domains. We find that active learning yields consistent gains across all SemEval 2021 Task 10 tasks and do- mains, but though the shared task saw success- ful self-trained and data augmented models, our systematic comparison finds these strate- gies to be unreliable for source-free domain adaptation. 1 Introduction Deep neural networks achieve high performance in many tasks, but typically require annotated training data for each new domain. Domain adaptation al- gorithms aim to take models trained on one domain (the “source domain”) and transfer the model’s knowledge to another domain (the “target domain”). They typically try to do this without a huge amount of annotated data in the target domain. Domain adaptation can be easy if the source and target do- main have similar distributions, but domains often differ substantially (Wilson and Cook, 2020). While there has been much progress in domain adaptation methods (Kouw, 2018) and even in un- supervised domain adaptation where there are no target-domain labels (Ramponi and Plank, 2020), most methods assume access to the labeled source data. Yet this assumption is often not satisfied, especially in the clinical domain due to privacy concerns (Laparra et al., 2020). SemEval 2021 Task 10 (Laparra et al., 2021), on source-free domain adaptation, called attention to this challenging but more realistic scenario where labeled source data are not accessible, only the model trained on the source domain data can be shared1, and little or no labeled target data are avail- able. Participants explored methods including self- training, active learning, and data augmentation (Laparra et al., 2021) but it is hard to make fair comparisons between algorithms since different teams varied in their base implementations. We therefore conducted experiments to provide a systematic comparison of algorithms for source- free domain adaptation. Our contributions are: 1. The first systematic comparison of self-training, active learning, and data augmentation for source-free domain adaptation, carried out across multiple tasks and domains. 2. We identify a formulation of source-free active learning that consistently improves performance of the source-domain model, and sometimes even outperforms fine-tuning on a large set of labeled target domain data. 3. We perform an error analysis across tasks and domains and show that the selected formulation of active learning corrects several types of errors that …

Content truncated to 500 words