Self-Adapter at SemEval-2021 Task 10: Entropy-based Pseudo-Labeler for Source-free Domain Adaptation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 452–457 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 452 Self-Adapter at SemEval-2021 Task 10: Entropy-based Pseudo-Labeler for Source-free Domain Adaptation Sangwon Yoon1,2, Yanghoon Kim1,3 and Kyomin Jung1,3 1Seoul National University, Seoul, Korea 2School of Law, Seoul National University, Seoul Korea 2Department of Electrical and Computer engineering, Seoul National University, Seoul Korea {sangwon38383, ad26kr, kjung}@snu.ac.kr Abstract Source-free domain adaptation is an emerging line of work in deep learning research since it is closely related to the real-world environ- ment. We study the domain adaption in the sequence labeling problem where the model trained on the source domain data is given. We propose two methods: Self-Adapter and Selec- tive Classifier Training. Self-Adapter is a train- ing method that uses sentence-level pseudo- labels filtered by the self-entropy threshold to provide supervision to the whole model. Selective Classifier Training uses token-level pseudo-labels and supervises only the clas- sification layer of the model. The pro- posed methods are evaluated on data provided by SemEval-2021 task 10 and Self-Adapter achieves 2nd rank performance. 1 Introduction Domain adaptation (DA) is the task of applying an algorithm trained on a source domain data to a different target domain data with limited/undefined labels. DA has gotten significant attention as an al- ternative of fine-tuning approach (Ganin and Lem- pitsky, 2015; Saito et al., 2018; Tzeng et al., 2017), especially in situations rich supervision is not possi- ble (Morerio et al., 2018). DA is an important way of overcoming the data shortage of deep learning since it enables the utilization of knowledge from other labeled data. Source-free DA is then proposed to cope with such data sharing in the general setting of DA, the data distribution in the source domain and the tar- get domain are related but different (Storkey and Sugiyama, 2007), and annotated samples from the source domain are available during the training pro- cess. However, many of the data resources are not allowed to be shared in real-life environments as there are increasing concerns for privacy issues. For example, Twitter has a regulation that prevents sharing tweet text. The policy is even more rig- orous in the financial/clinical domain under the privacy protection issue. Unlike conventional DA, one can not get ac- cess to the source domain data in source-free DA but is provided a model trained on the source do- main data. About source-free DA in computer vi- sion, several approaches have been proposed; (Sa- hoo et al., 2020) assumes the target domain data is a transformation from the source domain data along natural axes such as brightness and contrast; (Kundu et al., 2020) proposes universal DA that is trained via two-stage learning of procurement and deployment; (Kim et al., 2020) progressively updates the target model with pseudo-labels which are selected under self-entropy criterion. As for natural language processing (NLP), the application of source-free DA is slightly more com- plicated since sentences are usually considered as having discrete representations. In this …

Content truncated to 500 words