Named Entity Recognition in a Very Homogenous Domain

Findings of the Association for Computational Linguistics: EACL 2023, pages 1850–1855 May 2-6, 2023 ©2023 Association for Computational Linguistics Named Entity Recognition in a Very Homogeneous Domain Oshin Agarwal University of Pennsylvania oagarwal@seas.upenn.edu Ani Nenkova Adobe Research nenkova@adobe.com Abstract Machine Learning models have lower accuracy when tested on out-of-domain data. Developing models that perform well on several domains or can be quickly adapted to a new domain is an important research area. Domain, however, is a vague term, that can refer to any aspect of data such as language, genre, source and struc- ture. We consider a very homogeneous source of data, specifically sentences from news arti- cles from the same newspaper in English, and collect a dataset of such “in-domain” sentences annotated with named entities. We find that even in such a homogeneous domain, the per- formance of named entity recognition models varies significantly across news topics. Selec- tion of diverse data, as we demonstrate, is cru- cial even in a seemingly homogeneous domain. 1 Introduction Supervised neural models for named entity recog- nition achieve high accuracy when used in-domain. When models are evaluated or adapted (Daumé III, 2007; Wang et al., 2020; Gururangan et al., 2020) for out-of-domain text, or even developed for spe- cialized domains (Nguyen et al., 2020; Beltagy et al., 2019), the term domain generally refers to broad genres such as news, social media, or biomedical text. However, text can be (dis)similar in aspects beyond genre, such as the source of the data, its structure, or the time period. Dai et al. (2019) distinguish two aspects of domain— the genre and the tenor, which they describe as the participants in the discourse, their relationships and their purpose. They find that even though people consider genre to be more important for domain adaptation, tenor is important as well when select- ing pre-training data. The term domain encompasses more than just broadly defined genres. Online comments on differ- ent platforms can be considered different domains. So can news from different newspapers or differ- ent time periods. We show that even text from the same genre and source needs to be examined finely for topical or structural differences. We collect a dataset of news articles from the New York Times and annotate it for named entities. We find that the performance of NER models varies significantly even in this dataset when it is stratified based on news topics. While entities unseen in the training data can be a factor that contributes to performance degradation, we find that structural differences in sentences and entity ambiguity are the main con- tributors. Selecting diverse data is therefore crucial even in such “in-domain” settings. We show that even a very small number of sentences from each topic can help narrow the performance gap, and selecting random sentences rather than full docu- ments from the full corpus, will ensure that there is a good sample of diverse sentences. 2 Dataset The dataset is available at https://github.com/ oagarwal/nyt-ner. Here we describe the process of …

Content truncated to 500 words