Entity Enhanced BERT Pre-training for Chinese NER

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6384–6396, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6384 Entity Enhanced BERT Pre-training for Chinese NER Chen Jia†‡∗, Yuefeng Shi‡∗, Qinrong Yang3 and Yue Zhang‡§ †Fudan University, China ‡School of Engineering, Westlake University, China 3Tuiwen Technology Inc., China §Institute of Advanced Technology, Westlake Institute for Advanced Study, China {jiachen,shiyuefeng,zhangyue}@westlake.edu.cn yang bigarm@hotmail.com Abstract Character-level BERT pre-trained in Chinese suffers a limitation of lacking lexicon informa- tion, which shows effectiveness for Chinese NER. To integrate the lexicon into pre-trained LMs for Chinese NER, we investigate a semi- supervised entity enhanced BERT pre-training method. In particular, we first extract an entity lexicon from the relevant raw text using a new- word discovery method. We then integrate the entity information into BERT using Char- Entity-Transformer, which augments the self- attention using a combination of character and entity representations. In addition, an entity classification task helps inject the entity infor- mation into model parameters in pre-training. The pre-trained models are used for NER fine- tuning. Experiments on a news dataset and two datasets annotated by ourselves for NER in long-text show that our method is highly ef- fective and achieves the best results. 1 Introduction As a fundamental task in information extraction, named entity recognition (NER) is useful for NLP tasks such as relation extraction (Zelenko et al., 2003), event detection (Kumaran and Allan, 2004) and machine translation (Babych and Hartley, 2003). We investigate Chinese NER (Gao et al., 2005), for which the state-of-the-art methods use a character-based neural encoder augmented with lexicon word information (Zhang and Yang, 2018; Gui et al., 2019a,b; Xue et al., 2019). NER has been a challenging task due to the flex- ibility of named entities. There can be a large num- ber of OOV named entities in the open domain, which poses challenges to supervised learning al- gorithms. In addition, named entities can be am- biguous. Take Figure 1 for example. The term “老 妇人(the old lady)” literally means “older woman”. ∗Equal contribution. 老妇人(The old lady) (Juventus F.C.) wins nine consecutive Serie A titles Char-Entity-Transformer New-Word Discovery Raw Text ⊕ 老妇人 的队员进行了较大的调整，增加了许多后起之秀。 由于出现了"罗塞塔"事件，老妇人 被判罚6分，只取得北方区第5名。 ... Organization Organization ⊕ 冠 冠 连 九 九 连 甲 甲 意 意 取 取 豪 豪 人 人 妇 妇 老 老 ⊕ Figure 1: Entity enhanced pre-training for NER. “老妇 人(The old lady)”, the nickname of a football club Ju- ventus F.C., is extracted by new-word discovery and integrated into the Transformer structure. After pre- training, the embedding of “老妇人(The old lady)” has the global information and correctly classifies itself as an ORG, which also helps recognize “意甲(Serie A)” as an ORG. However, in the context of football news, it means the nickname of a football club Juventus F.C.. Thus entity lexicons that contain domain knowledge can be useful for the task (Radford et al., 2015; Xu et al., 2019). Intuitively, such lexicons can be collected auto- matically from a set of documents that are relevant to the …

Content truncated to 500 words