Noisy-Labeled NER with Confidence Estimation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3437–3445 June 6–11, 2021. ©2021 Association for Computational Linguistics 3437 Noisy-Labeled NER with Confidence Estimation Kun Liu1∗ Yao Fu2∗ Chuanqi Tan1† Mosha Chen1 Ningyu Zhang3 Songfang Huang1 Sheng Gao4 1Alibaba Group 2University of Edinburgh 3Zhejiang University 4 Guizhou Provincial Key Laboratory of Public Big Data, Guizhou University {kun.liu624; sheng.gao.81}@gmail.com yao.fu@ed.ac.uk {chuanqi.tcq; chenmosha.cms; songfang.hsf}@alibaba-inc.com zhangningyu@zju.edu.cn Abstract Recent studies in deep learning have shown significant progress in named entity recogni- tion (NER). Most existing works assume clean data annotation, yet a fundamental challenge in real-world scenarios is the large amount of noise from a variety of sources (e.g., pseudo, weak, or distant annotations). This work stud- ies NER under a noisy labeled setting with cali- brated confidence estimation. Based on empir- ical observations of different training dynam- ics of noisy and clean labels, we propose strate- gies for estimating confidence scores based on local and global independence assumptions. We partially marginalize out labels of low con- fidence with a CRF model. We further propose a calibration method for confidence scores based on the structure of entity labels. We in- tegrate our approach into a self-training frame- work for boosting performance. Experiments in general noisy settings with four languages and distantly labeled settings demonstrate the effectiveness of our method 1. 1 Introduction Recent progress in deep learning has significantly advanced NER performances (Lample et al., 2016; Devlin et al., 2018). While most existing works assume clean data annotation, real-world data in- evitably involve different levels of noise (e.g., dis- tant supervision from the dictionary (Peng et al., 2019), or weak supervision from the web Vran- deˇci´c and Krötzsch, 2014; Cao et al., 2019a). Fig- ure 1 gives an example of such noisy labels. To train robust models with high performance, it is fundamentally critical to tackle the challenges as- sociated with noisy data annotation. In this work, we propose a confidence estimation approach for NER with noisy labels. We motivate ∗ Equal Contribution. † Corresponding author. 1Our code can be found at https://github.com/ liukun95/Noisy-NER-Confidence-Estimation Brooklyn and Mary live in New York Gold Labels Noisy Labels B-LOC O B-PER O O O B-LOC B-PER O B-PER O O I-LOC B-LOC Figure 1: A noisy label example. Brooklyn and York are noisy positives. New is noisy negative. our approach with important empirical observa- tions of the training dynamics of clean and noisy labels: usually, clean data are easier to fit with faster convergence and smaller loss values (Jiang et al., 2018; Han et al., 2018a; Arazo et al., 2019). Consequently, loss values (probabilities or scores of labels) can serve as strong indicators for the existence of noise, which we utilize to build our confidence estimation. The key contribution of this work is a confidence estimation method with calibration. We use prob- abilities of labels as confidence scores and apply two estimation strategies based on global or local normalization that assume different dependency structures about how …

Content truncated to 500 words