Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7198 - 7211 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning Kang Zhou, Yuepei Li, Qi Li Department of Computer Science, Iowa State University, Ames, Iowa, USA {kangzhou, liyp0095, qli}@iastate.edu Abstract In this paper, we study the named entity recog- nition (NER) problem under distant supervi- sion. Due to the incompleteness of the exter- nal dictionaries and/or knowledge bases, such distantly annotated training data usually suffer from a high false negative rate. To this end, we formulate the Distantly Supervised NER (DS- NER) problem via Multi-class Positive and Un- labeled (MPU) learning and propose a theoreti- cally and practically novel CONFidence-based MPU (Conf-MPU) approach. To handle the incomplete annotations, Conf-MPU consists of two steps. First, a confidence score is esti- mated for each token of being an entity token. Then, the proposed Conf-MPU risk estimation is applied to train a multi-class classifier for the NER task. Thorough experiments on two benchmark datasets labeled by various external knowledge demonstrate the superiority of the proposed Conf-MPU over existing DS-NER methods. Our code is available at Github1. 1 Introduction Named Entity Recognition (NER) aims to detect entity mentions from text and classify them into predefined types. It is a fundamental task in in- formation extraction and many other downstream tasks (Gábor et al., 2018; Luan et al., 2017; Giorgi et al., 2019). However, the necessity of extensive human efforts to annotate a large amount of train- ing data imposes restrictions on the state-of-the-art supervised deep learning methods, especially in professional fields. To address this problem, distantly supervised methods spring up (Yang et al., 2018; Shang et al., 2018; Mayhew et al., 2019; Cao et al., 2019; Peng et al., 2019; Liang et al., 2020; Liu et al., 2021; Zhang et al., 2021), which aim to train NER models using automatically annotated training data based on external knowledge such as dictionaries and 1https://github.com/kangISU/Conf-MPU-DS-NER Figure 1: Distant labeling example for the entity type of Disease using a dictionary. knowledge bases. Observed by previous DS-NER methods (Shang et al., 2018; Peng et al., 2019), distant labels provided by reliable dictionaries are usually of high precision. However, such distant labeling suffers a major drawback — incomplete labeling. This is due to the fact that most exist- ing dictionaries and knowledge bases have limited coverage on entities. Hence simply treating unla- beled samples (i.e., unmatched tokens) as negative ones will introduce a high false negative rate (e.g., “neutropenia” in Figure 1) compared with human- annotated training data, and further mislead a su- pervised NER model to overfit to false negative samples and seriously affect its recall. Recently, binary Positive and Unlabeled (PU) learning is applied to DS-NER tasks for this chal- lenge (Peng et al., 2019). PU learning performs classification using only limited labeled positive data and unlabeled data, thus naturally suitable for handling distant supervision, where external knowledge …

Content truncated to 500 words