Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10367–10378 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10367 Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang, Yu Zhang, Heng Ji, Jiawei Han University of Illinois Urbana-Champaign, IL, USA {yumeng5, yzhan238, jiaxinh3, xwang174, yuz9, hengji, hanj}@illinois.edu Abstract We study the problem of training named en- tity recognition (NER) models using only distantly-labeled data, which can be automat- ically obtained by matching entity mentions in the raw text with entity types in a knowl- edge base. The biggest challenge of distantly- supervised NER is that the distant supervi- sion may induce incomplete and noisy labels, rendering the straightforward application of supervised learning ineffective. In this pa- per, we propose (1) a noise-robust learning scheme comprised of a new loss function and a noisy label removal step, for training NER models on distantly-labeled data, and (2) a self-training method that uses contextual- ized augmentations created by pre-trained lan- guage models to improve the generalization ability of the NER model. On three bench- mark datasets, our method achieves superior performance, outperforming existing distantly- supervised NER models by significant mar- gins1. 1 Introduction Named entity recognition (NER), which aims at identifying real-world entity mentions (e.g., per- son, location and organization names) from texts, is a fundamental task in natural language process- ing with a wide range of applications, including question answering (Khalid et al., 2008), knowl- edge base construction (Etzioni et al., 2005), text summarization (Aramaki et al., 2009) and dialog systems (Bowden et al., 2018). In recent years, deep neural models (Devlin et al., 2019; Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016) have achieved enormous success for NER, thanks to their strong representation learning power that accurately captures the entity semantics in tex- tual contexts. However, a common bottleneck of applying deep learning models is the acquisition 1Code can be found at https://github.com/ yumeng5/RoSTER. LOC PER ORG PER Distantly-Labeled Ground Truth PER PER Miguel Angel Jimenez is a professional golfer. Coopers and Lybrand emigrates to Basque Country for fiscal reasons. Miguel Angel Jimenez is a professional golfer. Coopers and Lybrand emigrates to Basque Country for fiscal reasons. Figure 1: Distant labels obtained with knowledge bases may be incomplete and noisy, resulting in wrongly- labeled tokens. of abundant high-quality human annotations, and this is especially the case for training NER models, which require every entity mention to be labeled in a sequence. To eliminate the need for human annotations, one direction is to use distant supervision for au- tomatic generation of entity labels. The common practice is to match entity mentions in the target corpus with typed entities in external gazetteers or knowledge bases. Unfortunately, such a distant labeling process inevitably introduces incomplete and noisy entity labels, because (1) the distant su- pervision source has limited coverage of the entity mentions in the target corpus, and (2) some entities can be matched to …

Content truncated to 500 words