Handling Rare Entities for Neural Sequence Labeling

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6441–6451 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6441 Handling Rare Entities for Neural Sequence Labeling Yangming Li♠,♥, Han Li♥, Kaisheng Yao♠ and Xiaolong Li♠ ♠Ant Financial Services Group, Alibaba Group ♥Harbin Institute of Technology {pangmao.lym,kaisheng.yao,xl.li}@antfin.com Abstract One great challenge in neural sequence label- ing is the data sparsity problem for rare entity words and phrases. Most of test set entities appear only few times and are even unseen in training corpus, yielding large number of out-of-vocabulary (OOV) and low-frequency (LF) entities during evaluation. In this work, we propose approaches to address this prob- lem. For OOV entities, we introduce local context reconstruction to implicitly incorpo- rate contextual information into their represen- tations. For LF entities, we present delex- icalized entity identification to explicitly ex- tract their frequency-agnostic and entity-type- specific representations. Extensive experi- ments on multiple benchmark datasets show that our model has significantly outperformed all previous methods and achieved new start- of-the-art results. Notably, our methods sur- pass the model fine-tuned on pre-trained lan- guage models without external resource. 1 Introduction In the context of natural language processing (NLP), the goal of sequence labeling is to assign a categorical label to each entity word or phrase in a text sequence. It is a fundamental area that under- lies a range of applications including slot filling and named entity recognition. Traditional methods use statistical models. Recent approaches have been based on neural networks (Collobert et al., 2011; Mesnil et al., 2014; Ma and Hovy, 2016; Strubell et al., 2017; Li et al., 2018; Devlin et al., 2018; Liu et al., 2019a; Luo et al., 2020; Xin et al., 2018) and they have made great progresses in various sequence labeling tasks. However, a great challenge to neural-network- based approaches is from the data sparsity problem (Augenstein et al., 2017). Specifically in the con- text of sequence labeling, the majority of entities Frequency Number Percentage = 0 (OOV) 1611 65.1% = 1 (Low) 191 7.7% < 10 (Low) 635 25.7% > 20 (High) 117 4.7% ≥ 0 (Total) 2475 100.0% Table 1: Number of occurrences of test set entities in the training set. OOV entities are those that have no occurrence (Frequency = 0) in the training set. Low frequency entities are those with fewer than ten occur- rences (Frequency < 10). Percentages of entity occur- rences are also shown. Data source is CoNLL-03. in test dataset may occur in training corpus few times or are absent at all. In this paper, we refer this phenomenon particularly to rare entity prob- lem. It is different from other types of data sparsity problems such as the lack of training data for low- resource language (Lin et al., 2018), as this rare entity problem is more related to a mismatch of en- tity distributions between training and test, rather than the size of training data. We present an exam- ple of the problem in Table 1. It …

Content truncated to 500 words