Evaluating Pretrained Transformer-based Models on the Task of Fine-Grained Named Entity Recognition

Proceedings of the 28th International Conference on Computational Linguistics, pages 3750–3760 Barcelona, Spain (Online), December 8-13, 2020 3750 Evaluating Pretrained Transformer-based Models on the Task of Fine-Grained Named Entity Recognition Cedric Lothritz University of Luxembourg cedric.lothritz@uni.lu Kevin Allix University of Luxembourg kevin.allix@uni.lu Lisa Veiber University of Luxembourg lisa.veiber@uni.lu Tegawend´e F. Bissyand´e University of Luxembourg tegawende.bissyande@uni.lu Jacques Klein University of Luxembourg jacques.klein@uni.lu Abstract Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task and has remained an active research field. In recent years, transformer models and more specifically the BERT model developed at Google revolutionised the field of NLP. While the performance of transformer-based approaches such as BERT has been studied for NER, there has not yet been a study for the fine-grained Named Entity Recognition (FG-NER) task. In this paper, we compare three transformer-based models (BERT, RoBERTa, and XLNet) to two non-transformer-based models (CRF and BiLSTM-CNN-CRF). Furthermore, we apply each model to a multitude of distinct domains. We find that transformer-based models incrementally outperform the studied non-transformer-based models in most domains with respect to the F1 score. Furthermore, we find that the choice of domain significantly influenced the performance regardless of the respec- tive data size or the model chosen. 1 Introduction Named Entity Recognition (NER) is part of the fundamental tasks in Natural Language Processing (NLP). The main objective of NER is to detect and classify proper names (named entities) in a free text. Typically, named entities can be subdivided into four broad categories: persons, i.e., first and last names, locations such as countries or landscapes, organisations such as companies or political parties, and miscellaneous entities which serves as a catch-all category for other named entities such as brands, meals, or social events. NER is an active research field and state-of-the-art solutions such as spaCy1, flair (Akbik et al., 2018), and Primer2 manage to achieve near-human performance. However, classical NER (which we refer to as coarse-grained NER in this paper) models typically distinguish between only a small number of entity types, usually fewer than a dozen distinct categories. While this kind of shallow classification is sufficient for many applications, there are industrial use- cases in which more precise information is necessary such as financial documents processing in the banking and finance context. For instance, application forms for a business loan are usually supplied with several supporting textual documents. These can contain the names of different types of persons, such as the owner or the CEO of the applying company, the contact person(s) at the issuing bank, finance analysts, or lawyers. The same is true for organisation names such as the name of the issuing bank, a government agency, or the name of the applying company or third-party companies. It is necessary to not only detect entity names, but to also qualify and differentiate between various entity types. Indeed, in many contexts the actual name of an entity is important only if it can be associated to a role, or any other relevant quality. In the banking and …

Content truncated to 500 words