Coarse-to-Fine Pre-training for Named Entity Recognition

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6345–6354, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6345 Coarse-to-Fine Pre-training for Named Entity Recognition Mengge Xue1,2 Bowen Yu1,2 Zhenyu Zhang1,2 Tingwen Liu1,2∗Yue Zhang3 and Bin Wang4 1Institute of Information Engineering, Chinese Academy of Sciences. Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences. Beijing, China 3School of Engineering, Westlake University, Hangzhou, China 4Xiaomi AI Lab, Xiaomi Inc., Beijing, China {xuemengge, yubowen, zhangzhenyu1996, liutingwen}@iie.ac.cn yue.zhang@wias.org.cn wangbin11@xiaomi.cn Abstract More recently, Named Entity Recognition has achieved great advances aided by pre-training approaches such as BERT. However, current pre-training techniques focus on building lan- guage modeling objectives to learn a gen- eral representation, ignoring the named entity- related knowledge. To this end, we propose a NER-specific pre-training framework to in- ject coarse-to-fine automatically mined entity knowledge into pre-trained models. Specifi- cally, we first warm-up the model via an en- tity span identification task by training it with Wikipedia anchors, which can be deemed as general-typed entities. Then we leverage the gazetteer-based distant supervision strategy to train the model extract coarse-grained typed entities. Finally, we devise a self-supervised auxiliary task to mine the fine-grained named entity knowledge via clustering. Empirical studies on three public NER datasets demon- strate that our framework achieves significant improvements against several pre-trained base- lines, establishing the new state-of-the-art per- formance on three benchmarks. Besides, we show that our framework gains promising re- sults without using human-labeled training data, demonstrating its effectiveness in label- few and low-resource scenarios.1 1 Introduction Named Entity Recognition (NER) is the task of dis- covering information entities and identifying their corresponding categories, such as mentions of peo- ple, organizations, locations, temporal and numeric expressions (Freitag, 2004). It is an essential com- ponent in many applications including machine translation (Babych and Hartley, 2003), relation ex- traction (Yu et al., 2019), entity linking (Xue et al., 2019a), and so on. ∗Corresponding Author 1The source code can be obtained from https://github.com/strawberryx/CoFEE Recently, NER has seen remarkable advances with the help of pre-trained representation mod- els, such as BERT (Devlin et al., 2019) and XL- Net (Yang et al., 2019). Providing contextual repre- sentation, these pre-trained models could be easily applied to NER applications as an encoder by just fine-tuning it. Despite refreshing the state-of-the- art performance of NER, the current pre-training techniques are not directly optimized for NER. Typ- ically, these models build unsupervised training objectives to capture dependency between words and learn a general language representation (Tian et al., 2020), while rarely considering incorporat- ing named entity information which can provide rich knowledge for NER. Due to little knowledge connection between NER and general language modeling, how to adapt public pre-trained models to be NER-specific remains an open problem. To this end, injecting named entity knowledge during pre-training is a possible solution. How- ever, this process of knowledge acquisition may be inefficient and expensive. In fact, there are ex- tensive weakly labeled annotations that naturally exist on the web yet …

Content truncated to 500 words