What do we really know about State of the Art NER?

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5983–5993 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5983 What do we Really Know about State of the Art NER? Sowmya Vajjala, Ramya Balasubramaniam National Research Council, Canada; Novisto.com sowmya.vajjala@nrc-cnrc.gc.ca, ramya.balasubramaniam@novisto.com Abstract Named Entity Recognition (NER) is a well researched NLP task and is widely used in real world NLP scenarios. NER research typically focuses on the creation of new ways of training NER, with relatively less emphasis on resources and evaluation. Further, state of the art (SOTA) NER models, trained on standard datasets, typically report only a single performance measure (F-score) and we don’t really know how well they do for different entity types and genres of text, or how robust are they to new, unseen entities. In this paper, we perform a broad evaluation of NER using a popular dataset, that takes into consideration various text genres and sources constituting the dataset at hand. Additionally, we generate six new adversarial test sets through small perturbations in the original test set, replacing select entities while retaining the context. We also train and test our models on randomly generated train/dev/test splits followed by an experiment where the models are trained on a select set of genres but tested genres not seen in training. These comprehensive evaluation strategies were performed using three SOTA NER models. Based on our results, we recommend some use- ful reporting practices for NER researchers, that could help in providing a better understanding of a SOTA model’s performance in future. Keywords: Named Entity Recognition, NER Evaluation, OntoNotes Dataset, English 1. Introduction NER is one of the most commonly researched infor- mation extraction tasks in NLP. It is also among the most common use-cases in industry, according to a re- cent survey (Lorica and Nathan, 2021). State of the art (SOTA) NER models report F-scores of over 90% on standard English datasets1. There are also several off-the-shelf NER tools that provide pre-trained models based on this research, which can be used in practical application scenarios. However, what do we really know about these NER models, beyond a single measure (typically, micro-F score)? One approach to answer this question could be to study how well does an NER model do on var- ious named entity categories and text genres, or how well does it generalize on data with previously un- seen genres and new test sets. Another approach could be to assess how robust the algorithms themselves are when trained and tested on different splits of the same dataset. There is some recent research in NLP that shows why reporting and comparing based on a sin- gle evaluation measure is not the best practice (Reimers and Gurevych, 2018; Dror et al., 2019). However, there isn’t much research focusing on NER in partic- ular. Consequently, to understand the performance of SOTA NER models in depth, and argue for better eval- uation approaches in future, we address the following research questions in …

Content truncated to 500 words