Exploring Cross-sentence Contexts for Named Entity Recognition with BERT

Proceedings of the 28th International Conference on Computational Linguistics, pages 904–914 Barcelona, Spain (Online), December 8-13, 2020 904 Exploring Cross-sentence Contexts for Named Entity Recognition with BERT Jouni Luoma TurkuNLP group University of Turku Turku, Finland jouni.a.luoma@utu.fi Sampo Pyysalo TurkuNLP group University of Turku Turku, Finland sampo.pyysalo@utu.fi Abstract Named entity recognition (NER) is frequently addressed as a sequence classification task with each input consisting of one sentence of text. It is nevertheless clear that useful information for NER is often found also elsewhere in text. Recent self-attention models like BERT can both capture long-distance relationships in input and represent inputs consisting of several sentences. This creates opportunities for adding cross-sentence information in natural language processing tasks. This paper presents a systematic study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context as additional sen- tences to BERT input systematically increases NER performance. Multiple sentences in input samples allows us to study the predictions of the sentences in different contexts. We propose a straightforward method, Contextual Majority Voting (CMV), to combine these different pre- dictions and demonstrate this to further increase NER performance. Evaluation on established datasets, including the CoNLL’02 and CoNLL’03 NER benchmarks, demonstrates that our pro- posed approach can improve on the state-of-the-art NER results on English, Dutch, and Finnish, achieves the best reported BERT-based results on German, and is on par with other BERT-based approaches in Spanish. We release all methods implemented in this work under open licenses. 1 Introduction Named entity recognition (NER) approaches have evolved through various methodological phases, broadly including rule/knowledge-based, unsupervised, feature engineering and supervised learning, and feature inferring approaches (Yadav and Bethard, 2018; Li et al., 2020a). The use of cross-sentence in- formation in some form has been a normal part of many NER methods in the former categories, but its role has diminished with the current feature inferring deep learning based approaches. Rule/knowledge- based approaches such as that of Mikheev et al. (1998) typically match strings to lexicons and similar domain knowledge sources, possibly going through text multiple times with refinement based on enti- ties found on earlier passes. Later, manually engineered features were used to incorporate information from the surrounding text, whole documents, data sets and also from external sources. The number of different features and classifiers grew during the years and it was normal that the features also contained cross-sentence information in some form as for example in (Krishnan and Manning, 2006). Dense repre- sentations of text such as word, character, string and subword embeddings first started to appear in NER methods as additional features given to classifiers (Collobert et al., 2011). Step by step, feature engi- neering has been demoted to a lesser role, as the most recent deep learning approaches learn to create meaningful and context-sensitive representations of text by pre-training with vast amounts of unlabelled data. These contextual representations are often used directly as features for existing NER architectures or fine-tuned with labelled data to match a …

Content truncated to 500 words