MedAI at SemEval-2021 Task 10: Negation-aware Pre-training for Source-free Negation Detection Domain Adaptation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1283–1288 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1283 MedAI at SemEval-2021 Task 10: Negation-aware Pre-training for Source-free Negation Detection Domain Adaptation Jinquan Sun, Qi Zhang, Yu Wang, Lei Zhang Alibaba Group Inc. {jinquan.sjq, mickey.zq, tonggou.wangyu}@alibaba-inc.com lei.zhang.lz@alibaba-inc.com Abstract Due to the increasing concerns for data privacy, source-free unsupervised domain adaptation at- tracts more and more research attention, where only a trained source model is assumed to be available, while the labeled source data remains private. To get promising adaptation results, we need to find effective ways to transfer knowl- edge learned in source domain and leverage useful domain specific information from tar- get domain at the same time. This paper de- scribes our winning contribution to SemEval 2021 Task 10: Source-Free Domain Adapta- tion for Semantic Processing. Our key idea is to leverage the model trained on source domain data to generate pseudo labels for target do- main samples. Besides, we propose Negation- aware Pre-training (NAP) to incorporate nega- tion knowledge into model. Our method wins the 1st place with F1-score of 0.822 on the offi- cial blind test set of Negation Detection Track. 1 Introduction The Negation Detection Track of SemEval 2021 Task 10: Source-Free Domain Adaptation for Se- mantic Processing provides a new setting for un- supervised domain adaptation task which ask par- ticipants to conduct negation detection in target domain only with model trained on source domain (namely source model) and unlabeled target do- main data. Negation detection is a span-in-context classification problem, where the model will jointly consider both the target mention to be classified and its surrounding context. For example, in sentence Has no <e> diarrhea <e\> and no new lumps or masses, the target span diarrhea is negated by its surrounding context no. This task is important for physicians to extract key information from clinical text. The test dataset used is based on MIMIC- III version 1.4 (Johnson et al., 2016), which is a large, freely-available english database comprising de-identified health-related data. We approach this task as a problem of learn- ing with pseudo labels. Our main interests in- clude 1) negation knowledge infusion through pre-training on target domain and 2) high-quality pseudo label generation. We divide the task into two stages: Negation-aware Pre-training (NAP) stage and Pseudo label Training stage. In the NAP stage, token-level and sentence-level negation se- mantic are embedded into model. In the pseudo label training stage, confidence threshold search and mean self-entropy are used to select target do- main samples with highly confident pseudo labels. 2 Related Work Traditional negation detection method are mostly rule-based. These methods (Chapman et al., 2001; Sanchez-Graillet and Poesio, 2007; Huang and Lowe, 2007; Sohn et al., 2012) used regular ex- pression algorithm, dependency parsing and gram- matical parsing to perform negation cue detection and scope resolution. Recent years, deep learning has been applied to negation detection task. In (Qian et al., 2016), Convolutional Neural Network was used to …

Content truncated to 500 words