Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss

Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss Thomas Effland Columbia University, USA teffland@cs.columbia.edu Michael Collins Google Research, USA mjcollins@google.com Abstract We study learning named entity recognizers in the presence of missing entity annotations. We approach this setting as tagging with la- tent variables and propose a novel loss, the Expected Entity Ratio, to learn models in the presence of systematically missing tags. We show that our approach is both theoretically sound and empirically useful. Experimentally, we find that it meets or exceeds performance of strong and state-of-the-art baselines across a variety of languages, annotation scenarios, and amounts of labeled data. In particular, we find that it significantly outperforms the previous state-of-the-art methods from Mayhew et al. (2019) and Li et al. (2021) by +12.7 and +2.3 F1 score in a challenging setting with only 1,000 biased annotations, averaged across 7 datasets. We also show that, when combined with our approach, a novel sparse annotation scheme outperforms exhaustive annotation for modest annotation budgets.1 1 Introduction Named entity recognition (NER) is a critical sub- task of many domain-specific natural language understanding tasks in NLP, such as information extraction, entity linking, semantic parsing, and question answering. For large, exhaustively anno- tated benchmark datasets, this problem has been largely solved by fine-tuning of high-capacity pre- trained sentence encoders from massive-scale lan- guage modeling tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019). However, fully anno- tated datasets themselves are expensive to obtain at scale, creating a barrier to rapid development of models in low-resource situations. Partial annotations, instead, may be much cheaper to obtain. For example, when building a 1We have published for our implementation and exper- imental results at https://github.com/teffland /ner-expected-entity-ratio. dataset for a new entity extraction task, a domain expert may be able to annotate entity spans with high precision at a lower recall by scanning through documents inexhaustively, creating a higher diversity of contexts and surface forms by limiting the amount of time spent on individual documents. In another scenario studied by Mayhew et al. (2019), non-speaker annotators for low- resource languages may only be able to recognize some of the more common entities in the target language, but will miss many less common ones. In both of these situations, we wish to leverage partially annotated training data with high preci- sion but low recall for entity spans. Because of the low recall, unannotated tokens are ambiguous and it is not reasonable to assume they are non-entities (the O tag). We give an example of this in Figure 1. We address the problem of training NER taggers with partially labeled, low-recall data by treating unannotated tags as latent variables for a discrim- inative tagging model. We propose to combine marginal tag likelihood training (Tsuboi et al., 2008) with a novel discriminative criterion, the Expected Entity Ratio (EER), to control the rela- tive proportion of entity tags in the sentence. The proposed loss is (1) flexibly able to incorporate prior knowledge about expected entity …

Content truncated to 500 words