PDALN: Progressive Domain Adaptation over a Pre-trained Model for Low-Resource Cross-Domain Named Entity Recognition

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5441–5451 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5441 PDALN: Progressive Domain Adaptation over a Pre-trained Model for Low-Resource Cross-Domain Named Entity Recognition Tao Zhang1, Congying Xia1, Philip S. Yu1, Zhiwei Liu1, Shu Zhao2 1University of Illinois at Chicago, Chicago, IL, USA; 2Anhui University, Hefei, Anhui, China {tzhang90,cxia8,psyu,zliu213}@uic.edu; zhaoshuzs2002@hotmail.com Abstract Cross-domain Named Entity Recognition (NER) transfers the NER knowledge from high-resource domains to the low-resource target domain. Due to limited labeled re- sources and domain shift, cross-domain NER is a challenging task. To address these chal- lenges, we propose a progressive domain adaptation Knowledge Distillation (KD) ap- proach – PDALN. It achieves superior domain adaptability by employing three components: (1) Adaptive data augmentation techniques, which alleviate cross-domain gap and label sparsity simultaneously; (2) Multi-level Do- main invariant features, derived from a multi- grained MMD (Maximum Mean Discrepancy) approach, to enable knowledge transfer across domains; (3) Advanced KD schema, which progressively enables powerful pre-trained lan- guage models to perform domain adaptation. Extensive experiments on four benchmarks show that PDALN can effectively adapt high- resource domains to low-resource target do- mains, even if they are diverse in terms and writing styles. Comparison with other base- lines indicates the state-of-the-art performance of PDALN. 1 Introduction Named Entity Recognition (NER) is typically framed as a sequence labeling task that targets to locate and classify named entities in text into prede- fined semantic types, such as Person, Organization, Location, etc. NER is a fundamental task in in- formation extraction (Karatay and Karagoz, 2015) and text understanding (Krasnashchok and Jouili, 2018). The effectiveness of most existing NER models depends on sufficient labeled data, which is time-consuming and labor-intensive. Current re- search proposes cross-domain NER, which enables NER on the low-resource target domain by trans- ferring knowledge from other high-resource source domains. However, it is challenging to build a cross- domain NER component with high precision and recall, due to the domain shift problem (Ben-David et al., 2010). When casting the cross-domain NER as a transfer learning problem, most solutions (He and Sun, 2017; Yang et al., 2017; Aguilar et al., 2017; Lee et al., 2018; Liu et al., 2020b) require high-quality cross-domain features for knowledge transfer. Limited labeled data prohibit transfer learning from extracting informative features. Be- sides, it is hard to find a single training dataset covering all the required NER types. Even if words overlap across domains, their combination or usage is different from each other. Domain adaptation (Sun et al., 2015) is widely studied to solve the domain shift issue. Existing approaches mainly introduce either word-level or discourse-level domain adaptations to enable cross- domain NER. To mitigate the word-level discrep- ancy, previous endeavors propose distributed word embedding (Kulkarni et al., 2016), label-aware maximum mean discrepancy estimation (Wang et al., 2018), and projecting learning (Lin and Lu, 2018). As to the discourse-level discrepancy, ex- isting approaches introduce multi-level adaptation layers (Lin and Lu, 2018), tensor decomposition (Jia et al., 2019), …

Content truncated to 500 words