Toward Fully Exploiting Heterogeneous Corpus:A Decoupled Named Entity Recognition Model with Two-stage Training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1641–1652 August 1–6, 2021. ©2021 Association for Computational Linguistics 1641 Toward Fully Exploiting Heterogeneous Corpus: A Decoupled Named Entity Recognition Model with Two-stage Training Yun Hu12 ∗, Yeshuang Zhu3, Jinchao Zhang3, Changwen Zheng12, Jie Zhou3 1 Institute of Software, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Pattern Recognition Center, WeChat AI, Tencent Inc, China {yunhu2016,changwen}@iscas.ac.cn {yshzhu,dayerzhang,withtomzhou}@tencent.com Abstract Named Entity Recognition (NER) is a fun- damental and widely used task in natural language processing (NLP), which is gener- ally trained on the human-annotated corpus. However, data annotation is costly and time- consuming, which restricts its scale and further leads to the performance bottleneck of NER models. In reality, we can conveniently collect large-scale entity dictionaries and distantly su- pervised data. However, the collected dictio- naries are lack of semantic context and the distantly supervised training instances contain large noise, which will bring uncertain effects to NER models when directly incorporated into the high-quality training set. To address the above issue, we propose a BERT-based de- coupled NER model with two-stage training to appropriately take advantage of the heteroge- neous corpus, including dictionaries, distantly supervised instances, and human-annotated in- stances. Our decoupled model consists of a Mention-BERT and a Context-BERT to re- spectively learn from the context-deficient dic- tionaries and noised distantly supervised in- stances at the pre-training stage. At the unified- training stage, the two BERTs are trained to- gether on human-annotated data to predict the correct labels for candidate regions. Empirical studies on three Chinese NER datasets demon- strate that our method achieves significant im- provements against several baselines, estab- lishing the new state-of-the-art performance. 1 Introduction Named entity recognition is a fundamental Natural Language Processing task that labels each word in sentences with predefined types, such as Per- son (PER), Location (LOC), Organization (ORG), etc. The results of NER can be used in many ∗ Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc. Work was done when Yun Hu was intern at WeChat AI. downstream NLP tasks, e.g., relation extraction (Bunescu and Mooney, 2005), information retrieval (Chen et al., 2015), and question answering (Yao and Van Durme, 2014). Supervised methods are mainstream approaches to NER, including CRF (Lafferty et al., 2001) and neural network models (Collobert et al., 2011; Lample et al., 2016; Ma and Hovy, 2016). Recently, large-scale pre-trained language models fine-tuned upon a limited amount of annotated data achieve competitive or better per- formance in NER task (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019b). Supervised NER methods require a sufficient amount of sentence-level annotated data, even for the methods using pre-trained language mod- els. However, obtaining sentence-level annotated data is expensive and thus leads to small train- ing data size and performance bottleneck of su- pervised models. In practice, entity dictionaries (or gazetteers) and unlabeled corpora can be ob- tained at a low cost. Furthermore, distantly su- pervised data can be automatically generated by matching …

Content truncated to 500 words