Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1775–1789 August 1–6, 2021. ©2021 Association for Computational Linguistics 1775 Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data Haoming Jiang∗ 1 , Danqing Zhang2, Tianyu Cao2, Bing Yin2, Tuo Zhao1 1Georgia Institute of Technology, Atlanta, GA, USA 2Amazon.com Inc, Palo Alto, CA, USA {jianghm, tourzhao}@gatech.edu {danqinz, caoty, alexbyin}@amazon.com Abstract Weak supervision has shown promising results in many natural language processing tasks, such as Named Entity Recognition (NER). Ex- isting work mainly focuses on learning deep NER models only with weak supervision, i.e., without any human annotation, and shows that by merely using weakly labeled data, one can achieve good performance, though still under- performs fully supervised NER with manu- ally/strongly labeled data. In this paper, we consider a more practical scenario, where we have both a small amount of strongly labeled data and a large amount of weakly labeled data. Unfortunately, we observe that weakly labeled data does not necessarily improve, or even deteriorate the model performance (due to the extensive noise in the weak labels) when we train deep NER models over a simple or weighted combination of the strongly labeled and weakly labeled data. To address this is- sue, we propose a new multi-stage computa- tional framework – NEEDLE with three essen- tial ingredients: (1) weak label completion, (2) noise-aware loss function, and (3) final fine- tuning over the strongly labeled data. Through experiments on E-commerce query NER and Biomedical NER, we demonstrate that NEE- DLE can effectively suppress the noise of the weak labels and outperforms existing methods. In particular, we achieve new SOTA F1-scores on 3 Biomedical NER datasets: BC5CDR- chem 93.74, BC5CDR-disease 90.69, NCBI- disease 92.28. 1 Introduction Named Entity Recognition (NER) is the task of detecting mentions of real-world entities from text and classifying them into predefined types. For example, the task of E-commerce query NER is to identify the product types, brands, product at- tributes of a given query. Traditional deep learning ∗ Work was done during internship at Amazon. approaches mainly train the model from scratch (Ma and Hovy, 2016; Huang et al., 2015), and rely on large amounts of labeled training data. As NER tasks require token-level labels, annotating a large number of documents can be expensive, time- consuming, and prone to human errors. Therefore, the labeled NER data is often limited in many do- mains (Leaman and Gonzalez, 2008). This has become one of the biggest bottlenecks that pre- vent deep learning models from being adopted in domain-specific NER tasks. To achieve better performance with limited la- beled data, researchers resort to large unlabeled data. For example, Devlin et al. (2019) propose to pre-train the model using masked language mod- eling on large unlabeled open-domain data, which is usually hundreds/thousands of times larger than the manually/strongly labeled data. However, open- domain pre-trained models can only provide lim- ited semantic and syntax information for …

Content truncated to 500 words