HardEval: Focusing on Challenging Tokens to Assess Robustness of NER

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1704–1711 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1704 HardEval: Focusing on Challenging Tokens to Assess Robustness of NER Gabriel Bernier-Colborne, Philippe Langlais NRC, RALI-DIRO Montreal, Canada gabriel.bernier-colborne@nrc.ca, felipe@iro.umontreal.ca Abstract To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity. These subsets provide a system-agnostic basis for evaluating specific sources of NER errors and assessing room for improvement in terms of robustness. We analyze these subsets of challenging tokens in two widely-used NER benchmarks, then exploit them to evaluate NER systems in both in-domain and out-of-domain settings. Results show that these challenging tokens explain the majority of errors made by modern NER systems, although they represent only a small fraction of test tokens. They also indicate that label shift is harder to deal with than unknown words, and that there is much more room for improvement than the standard NER evaluation procedure would suggest. We hope this work will encourage NLP researchers to adopt rigorous and meaningful evaluation methods, and will help them develop more robust models. Keywords: named entity recognition, natural language processing, evaluation, robustness 1. Introduction Named entity recognition (NER) is one of the most com- mon applications of natural language processing (NLP). Given a text, an NER system is tasked with detecting ex- pressions that refer to named entities, e.g. people, locations or organizations, and predicting the entity type of each de- tected mention. NER is used in various downstream appli- cations, which typically involve information extraction or retrieval. NER is a relatively well-studied problem, and is often used to benchmark NLP systems. The standard method used to assess NER systems is an automatic, quantitative evaluation based on a human- annotated dataset, whereby the system’s output is compared to the human annotations. Most of the time, systems are trained and tested on data from the same domain and distri- bution, i.e. disjoint subsets of the same dataset, typically a fixed train/dev/test split. The high scores achieved by modern NLP systems in this setting suggest that their per- formance is close to, or even better than, human perfor- mance on this task. Thus, one might conclude that NER is a “solved problem”. However, cross-domain evaluation, whereby systems are trained and tested on data from different domains, paints a less rosy picture, as the change in the data distribution results in poorer performance. This suggests that NER sys- tems lack robustness, and have a limited ability to learn what a named entity actually is in a way that generalizes across domains. To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity. These sources of errors arise more frequently when there is a shift in the data distribution, but …

Content truncated to 500 words