Named Entity Recognition - Is There a Glass Ceiling?

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 624–633 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 624 Named Entity Recognition - Is there a glass ceiling? Tomasz Stanislawek†,‡, Anna Wr´oblewska†,‡, Alicja W´ojcika†,§, Daniel Ziembicki†,§ Przemyslaw Biecek‡,¶ †Applica.ai, Warsaw, Poland ¶Samsung Research Poland, Warsaw, Poland ‡Faculty of Mathematics and Information Science, Warsaw University of Technology §Department of Formal Linguistics, University of Warsaw Abstract Recent developments in Named Entity Recog- nition (NER) have resulted in better and bet- ter models. However, is there a glass ceil- ing? Do we know which types of errors are still hard or even impossible to correct? In this paper, we present a detailed analysis of the types of errors in state-of-the-art machine learning (ML) methods. Our study reveals the weak and strong points of the Stanford, CMU, FLAIR, ELMO and BERT models, as well as their shared limitations. We also introduce new techniques for improving annotation, for training processes and for checking a model’s quality and stability. Presented results are based on the CoNLL 2003 data set for the English language. A new enriched semantic annotation of errors for this data set and new diagnostic data sets are at- tached in the supplementary materials. 1 Introduction The problem of Named Entity Recognition (NER) was defined over 20 years ago at the Message Un- derstanding Conference (MUC, 1995; Sundheim, 1995). Nowadays, there are a lot of solutions ca- pable of a very high accuracy even on very hard and multi-domain data sets (Yadav and Bethard, 2018; Li et al., 2018). Many of these solutions benefit from large available data sets or from recent developments in deep neural networks. However, in order to progress further with this last mile, we need a bet- ter understanding of the sources of errors in NER problem; as it is stated that ”The first step to ad- dress any problem is to understand it”. We per- formed a detailed analysis of errors on the pop- ular CoNLL 2003 data set (Tjong Kim Sang and De Meulder, 2003). Of course, different models make different mis- takes. Here, we have focused on models that con- stitute a kind of breakthrough in the NER do- main. These models are: Stanford NER (Finkel et al., 2005), the model made by the NLP team from Carnegie Mellon University (CMU) (Lample et al., 2016), ELMO (Peters et al., 2018), FLAIR (Akbik et al., 2018) and BERT-Base (Devlin et al., 2018). In the Stanford model, Conditional Ran- dom Fields (CRF) with manually created features were tackled. Lample and the team (at CMU) used an LSTM deep neural network with an output with CRF for the first time. ELMO and FLAIR are new language modeling techniques as an encoder, and LSTM with a CRF layer as an output decoder. A team from Google used a fine-tuning approach with the BERT model in a NER problem for the first time, based on a Bi-diREctional Transformer language model (LM). We analyzed the data set from …

Content truncated to 500 words