CrossWeigh: Training Named Entity Tagger from Imperfect Annotations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5154–5163, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5154 CrossWeigh: Training Named Entity Tagger from Imperfect Annotations Zihan Wang∗ Jingbo Shang∗ Liyuan Liu∗ Lihao Lu Jiacheng Liu University of Illinois at Urbana-Champaign, Urbana, IL, USAJiawei Han {zihanw2, shang7, ll2, lihaolu2, jl25, hanj}@illinois.edu Abstract Everyone makes mistakes. So do human an- notators when curating labels for named entity recognition (NER). Such label mistakes might hurt model training and interfere model com- parison. In this study, we dive deep into one of the widely-adopted NER benchmark datasets, CoNLL03 NER. We are able to identify la- bel mistakes in about 5.38% test sentences, which is a significant ratio considering that the state-of-the-art test F1 score is already around 93%. Therefore, we manually correct these la- bel mistakes and form a cleaner test set. Our re-evaluation of popular models on this cor- rected test set leads to more accurate assess- ments, compared to those on the original test set. More importantly, we propose a simple yet effective framework, CrossWeigh, to han- dle label mistakes during NER model training. Specifically, it partitions the training data into several folds and train independent NER mod- els to identify potential mistakes in each fold. Then it adjusts the weights of training data ac- cordingly to train the final NER model. Exten- sive experiments demonstrate significant im- provements of plugging various NER models into our proposed framework on three datasets. All implementations and corrected test set are available at our Github repo 1. 1 Introduction Named entity recognition (NER), identifying both spans and types of named entities in text, is a fundamental task in the natural language process- ing pipeline. On one of the widely-adopted NER benchmarks, the CoNLL03 NER dataset (Sang and Meulder, 2003), the state-of-the-art NER per- formance has been pushed to a F1 score around 93% (Akbik et al., 2019), through building end-to- end neural models (Lample et al., 2016; Ma and ∗Equal Contributions. 1https://github.com/ZihanWangKi/ CrossWeigh Hovy, 2016) and introducing language models for contextualized representations (Peters et al., 2017, 2018; Akbik et al., 2018; Liu et al., 2018a). Such high performance makes the label mistakes in man- ually curated “gold standard” data non-negligible. For example, given a sentence “Chicago won game 1 with Derrick Rose scoring 25 points.”, this “Chicago”, representing the NBA team Chicago Bulls, should be annotated as an organization. However, when annotators are not careful or lack background knowledge, this “Chicago” might be annotated as a location, thus being a label mistake. These label mistakes bring up two challenges to NER: (1) mistakes in the test set can interfere the evaluation results and even lead to an inaccurate assessment of model performance; and (2) mis- takes in the training set can hurt NER model train- ing. Therefore, in this paper, we conduct empirical studies to understand these mistakes, correct the mistakes in the test set to …

Content truncated to 500 words