MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5590 - 5600 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective Xiao Wang⋆, Shihan Dou⋆, Limao Xiong⋆, Yicheng Zou⋆, Qi Zhang⋆,♣∗, Tao Gui♦, Liang Qiao♠, Zhanzhan Cheng♠, Xuanjing Huang⋆ ⋆ School of Computer Science, Fudan University, Shanghai, China ♦ Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China ♣ Shanghai Key Laboratory of Intelligent Information Processing, Shanghai, China ♠ Hikvision Research Institute, Hangzhou, China {xiao_wang20,yczou18,qz,tgui,xjhuang}@fudan.edu.cn {shdou21,lmxiong21}@m.fudan.edu.cn Abstract NER model has achieved promising perfor- mance on standard NER benchmarks. How- ever, recent studies show that previous ap- proaches may over-rely on entity mention in- formation, resulting in poor performance on out-of-vocabulary (OOV) entity recognition. In this work, we propose MINER, a novel NER learning framework, to remedy this is- sue from an information-theoretic perspective. The proposed approach contains two mutual information-based training objectives: i) gen- eralizing information maximization, which en- hances representation via deep understanding of context and entity surface forms; ii) super- fluous information minimization, which dis- courages representation from rote memorizing entity names or exploiting biased cues in data. Experiments on various settings and datasets demonstrate that it achieves better performance in predicting OOV entities. 1 Introduction Named Entity Recognition (NER) aims to identify and classify entity mentions from unstructured text, e.g., extracting location mention "Berlin" from the sentence "Berlin is wonderful in the winter". NER is a key component in information retrieval (Tan et al., 2021), question answering (Min et al., 2021), dialog systems (Wang et al., 2020), etc. Traditional NER models are feature-engineering and machine learning based (Zhou and Su, 2002; Takeuchi and Collier, 2002; Agerri and Rigau, 2016). Benefiting from the development of deep learning, neural- network-based NER models have achieved state- of-the-art results on several public benchmarks (Lample et al., 2016; Peters et al., 2018; Devlin et al., 2018; Yamada et al., 2020; Yan et al., 2021). Recent studies (Lin et al., 2020; Agarwal et al., 2021) show that, context does influence predictions ∗Corresponding authors. Precision Recall InDict OutDict Diff InDict OutDict Diff PER 88.03 75.40 14% 92.90 85.20 8% ORG 73.51 72.77 1% 81.93 76.56 7% GPE 79.55 78.21 2% 85.37 77.22 10% FAC 65.91 65.67 0% 86.05 65.67 24% ALL 83.37 71.97 12% 89.08 79.11 11% Table 1: The comparison between the in-dictionary and out-of-dictionary parts of the CoNLL 2003 baseline (Lin et al., 2020), which was tested on Bert-CRF. It is obvious that the performance gap between InDict and OutDict is significantly large. of NER models, but the main factor driving high performance is learning the named tokens them- selves. Consequently, NER models underperform when predicting entities that have not been seen during training (Fu et al., 2020; Lin et al., 2020), which is referred to as an Out-of-Vocabulary (OOV) problem. There are three classical strategies to alleviate the OOV problem: external knowledge, OOV word embedding, and contextualized embedding. The first one is …

Content truncated to 500 words