Identifying Incorrect Labels in the CoNLL-2003 Corpus

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 215–226 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 215 Identifying Incorrect Labels in the CoNLL-2003 Corpus Frederick Reiss1,2, Hong Xu2∗, Bryan Cutler2∗, Karthik Muthuraman2∗and Zachary Eichenberger1,3∗ 1IBM Research – Almaden, San Jose, CA 95120, USA 2IBM Center for Open Source Data and AI Technologies (CODAIT), San Francisco, CA 94105, USA 3University of Michigan, Ann Arbor, MI 48109, USA frreiss@us.ibm.com, hongx@ibm.com, bjcutler@us.ibm.com karthik.muthuraman@ibm.com, zachary.eichen@gmail.com Abstract The CoNLL-2003 corpus for English- language named entity recognition (NER) is one of the most influential corpora for NER model research. A large number of publica- tions, including many landmark works, have used this corpus as a source of ground truth for NER tasks. In this paper, we examine this corpus and identify over 1300 incorrect labels (out of 35089 in the corpus). In particular, the number of incorrect labels in the test fold is comparable to the number of errors that state-of-the-art models make when running inference over this corpus. We describe the process by which we identi- fied these incorrect labels, using novel variants of techniques from semi-supervised learning. We also summarize the types of errors that we found, and we revisit several recent results in NER in light of the corrected data. Finally, we show experimentally that our corrections to the corpus have a positive impact on three state-of- the-art models. 1 Introduction The English-language portion of the CoNLL-2003 shared task (Tjong Kim Sang and De Meulder, 2003) (henceforth CoNLL-2003) is one of the most widely-used benchmarks for named entity recog- nition (NER) models. It consists of news articles from the Reuters RCV1 corpus (Lewis et al., 2004). Since its debut, CoNLL-2003 has played a cen- tral role in NLP research. Over 2300 research papers have cited the original CoNLL-2003 pa- per1. Among these works, many are landmark results that have revolutionized the field of nat- ural language processing, including Glove embed- dings (Pennington et al., 2014), BERT embed- dings (Devlin et al., 2019), conditional random ∗The last four authors have contributed equally. 1https://scholar.google.com/scholar? cites=17103810098319730115 fields (Sutton and McCallum, 2012), and bidirec- tional LSTM models (Lample et al., 2016). The CoNLL-2003 corpus continues to be used in NER research. The Papers with Code website (Pa- per with Code, 2020), which tracks state-of-the-art F1 scores2 for this corpus, currently (as of July 2020) shows 43 results from 2016 through 2019 that improved this metric. While researchers have relied heavily on the CoNLL-2003 corpus as a source of ground truth, few have paid attention to the corpus itself. Er- rors in the corpus could potentially mislead and even divert the course of future research. Recent work has pointed out that improper benchmarking can have significant impact on evaluating machine learning algorithms (Smith-Miles et al., 2014). The fact that Stanislawek et al. (2019) and Wang et al. (2019) found many errors while examining parts of the corpus is even more alarming. A detailed examination of the corpus has become imperative. In this …

Content truncated to 500 words