Rethinking Negative Sampling for Handling Missing Entity Annotations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7188 - 7197 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Rethinking Negative Sampling for Handling Missing Entity Annotations Yangming Li, Lemao Liu, Shuming Shi Tencent AI Lab Shenzhen, China {newmanli,redmondliu,shumingshi}@tencent.com Abstract Negative sampling is highly effective in han- dling missing annotations for named entity recognition (NER). One of our contributions is an analysis on how it makes sense through introducing two insightful concepts: missam- pling and uncertainty. Empirical studies show low missampling rate and high uncertainty are both essential for achieving promising perfor- mances with negative sampling. Based on the sparsity of named entities, we also theoreti- cally derive a lower bound for the probability of zero missampling rate, which is only rele- vant to sentence length. The other contribu- tion is an adaptive and weighted sampling dis- tribution that further improves negative sam- pling via our former analysis. Experiments on synthetic datasets and well-annotated datasets (e.g., CoNLL-2003) show that our proposed approach benefits negative sampling in terms of F1 score and loss convergence. Besides, models with improved negative sampling have achieved new state-of-the-art results on real- world datasets (e.g., EC). 1 Introduction With powerful neural networks and abundant well-labeled corpora, named entity recognition (NER) models have achieved promising perfor- mances (Huang et al., 2015; Ma and Hovy, 2016; Akbik et al., 2018; Li et al., 2020a). However, in many scenarios, available training data is low- quality, which means a portion of named entities are absent in annotations. Fig. 1 depicts a sen- tence and its incomplete annotations. Fine-grained NER (Ling and Weld, 2012) is a typical case. Its training data is mainly obtained through apply- ing weak supervision to unlabeled corpora. Past works (Shang et al., 2018b; Li et al., 2021a) find missing annotations impact NER models and refer this to unlabeled entity problem. Recently, Li et al. (2021a) find it’s the misguid- ance of unlabeled entities to NER models in train- Japan , co-hosts of the World Cup in 2002 and ranked 20th in the world by FIFA , are favourites to regain their title here . LOC MISC ORG TIME QUANTITY Figure 1: A toy example to show unlabeled entity prob- lem. The phrases underlined with dashed lines are the named entities neglected by annotators. ing that causes their poor performances. To elim- inate this adverse impact, they propose a simple yet effective approach based on negative sampling. Compared with its counterparts (Li and Liu, 2005; Tsuboi et al., 2008; Shang et al., 2018b; Peng et al., 2019), this method is of high flexibility, without relying on external resources, heuristics, etc. While negative sampling has handled missing annotations well, there is no systematic study on how it works, especially what potential factors are involved. From a number of experiments, we find missampling and uncertainty both worth receiving attention. Missampling means that some unlabeled entities are mistakenly drawn into the set of train- ing negatives by negative sampling. To quantitively …

Content truncated to 500 words