SemEval-2021 Task 10: Source-Free Domain Adaptation for Semantic Processing

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 348–356 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 348 SemEval-2021 Task 10: Source-Free Domain Adaptation for Semantic Processing Egoitz Laparra1 Xin Su1 Yiyun Zhao1 ¨Ozlem Uzuner2 Timothy A Miller3 Steven Bethard1 1University of Arizona, Tucson, AZ 85721, USA {laparra, xinsu, yiyunzhao, bethard}@email.arizona.edu 2George Mason University, Fairfax, VA 22030, USA ouzuner@gmu.edu 3Boston Children’s Hospital and Harvard Medical School, Boston, MA 02115, USA timothy.miller@childrens.harvard.edu Abstract This paper presents the Source-Free Domain Adaptation shared task held within SemEval- 2021. The aim of the task was to explore adap- tation of machine-learning models in the face of data sharing constraints. Specifically, we consider the scenario where annotations exist for a domain but cannot be shared. Instead, participants are provided with models trained on that (source) data. Participants also receive some labeled data from a new (development) domain on which to explore domain adapta- tion algorithms. Participants are then tested on data representing a new (target) domain. We explored this scenario with two different se- mantic tasks: negation detection (a text classi- fication task) and time expression recognition (a sequence tagging task). 1 Introduction Data sharing restrictions are common in NLP datasets. For example, Twitter policies do not al- low sharing of tweet text, though tweet IDs may be shared. The situation is even more common in clinical NLP, where patient health information must be protected, and annotations over health text, when released at all, often require the signing of complex data use agreements. The Source-Free Domain Adaptation shared task presents a new framework that asks participants to develop semantic annotation systems in the face of data sharing constraints. A participant’s goal is to develop an accurate system for a target domain when annotations exist for a related domain but cannot be distributed. Instead of annotated training data, participants are given a model trained on the annotations. Then, given unlabeled target domain data, they are asked to make predictions. This is a challenging setting, and much previous work on domain adaptation does not apply, as it assumes access to source data (Ganin et al., 2016; Ziser and Reichart, 2017; Saito et al., 2017; Ruder and Plank, 2018), or assumes that labeled target domain data is available (Daum´e III, 2007; Xia et al., 2013; Kim et al., 2016; Peng and Dredze, 2017). Two different semantic tasks in English were cre- ated to explore this framework: negation detection and time expression recognition. These represent two common types of classification tasks: negation detection is typically formulated as predicting an attribute of a word or span given its context, and time expression recognition is typically formulated as a named entity tagging problem. Both of these tasks have previously been run as shared tasks, and had at least two different domains of data available, and we had access to experienced annotators for both tasks, allowing us to annotate data in a new domain. Negation detection is the task of identifying negation cues in text. This …

Content truncated to 500 words