Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 309–319 November 20–23, 2022. ©2022 Association for Computational Linguistics 309 Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts Asahi Ushio1, Leonardo Neves2, Vítor Silva2, Francesco Barbieri2, Jose Camacho-Collados1 1Cardiff NLP, School of Computer Science and Informatics, Cardiff University, United Kingdom {UshioA,CamachoColladosJ}@cardiff.ac.uk 2Snap Inc., Santa Monica, CA, United States {lneves,vsilvasousa,fbarbieri}@snap.com Abstract Recent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well- formatted documents such as news, Wikipedia, or scientific articles. In social media the land- scape is different, in which it adds another layer of complexity due to its noisy and dynamic na- ture. In this paper, we focus on NER in Twit- ter, one of the largest social media platforms, and construct a new NER dataset, TweetNER7, which contains seven entity types annotated over 11,382 tweets from September 2019 to August 2021. The dataset was constructed by carefully distributing the tweets over time and taking representative trends as a basis. Along with the dataset, we provide a set of language model baselines and perform an analysis on the language model performance on the task, espe- cially analyzing the impact of different time pe- riods. In particular, we focus on three important temporal aspects in our analysis: short-term degradation of NER models over time, strate- gies to fine-tune a language model over differ- ent periods, and self-labeling as an alternative to lack of recently-labeled data. TweetNER7 is released publicly1 along with the models fine- tuned on it2. 1 Introduction Named Entity Recognition (NER) is a long- standing NLP task that consists of identifying an entity in a sentence or document, and classifying it into an entity-type from a fixed typeset. One of the most common and successful types of NER sys- tem is achieved by fine-tuning pre-trained language models (LMs) on a human-annotated NER dataset 1https://huggingface.co/datasets/tner/ tweetner7 2NER models have been integrated into TweetNLP (Camacho-Collados et al., 2022) and can be found at https://github.com/asahi417/tner/tree/master/ examples/tweetner7_paper with token-wise classification (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018, 2019; Devlin et al., 2019). Remarkably, LM fine- tuning based NER models (Yamada et al., 2020; Li et al., 2020) already achieve over 90% F1 score in standard NER datasets such as CoNLL2003 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes5 (Hovy et al., 2006). However, NER is far from being solved, specialized domains such as financial news (Salinas Alvarado et al., 2015), biochemi- cal (Collier and Kim, 2004), or biomedical (Wei et al., 2015; Li et al., 2016) still pose additional challenges (Ushio and Camacho-Collados, 2021). Lower performance in these domains may be at- tributed to various factors such as the usage specific terminologies within those domains, which LMs have not seen while pre-training (Lee et al., 2020). Among …

Content truncated to 500 words