Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7199–7208 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7199 Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis Hang Jiang, Yining Hua, Doug Beeferman, Deb Roy MIT Center for Constructive Communication 75 Amherst St, Cambridge, MA 02139 {hjian42, ninghua, dougb5, dkroy}@mit.edu Abstract Social media data such as Twitter messages (“tweets”) pose a particular challenge to NLP systems because of their short, noisy, and colloquial nature. Tasks such as Named Entity Recognition (NER) and syntactic parsing require highly domain-matched training data for good performance. To date, there is no complete training corpus for both NER and syntactic analysis (e.g., part of speech tagging, dependency parsing) of tweets. While there are some publicly available annotated NLP datasets of tweets, they are only designed for individual tasks. In this study, we aim to create Tweebank-NER, an English NER corpus based on Tweebank V2 (TB2), train state-of-the-art (SOTA) Tweet NLP models on TB2, and release an NLP pipeline called Twitter-Stanza. We annotate named entities in TB2 using Amazon Mechanical Turk and measure the quality of our annotations. We train the Stanza pipeline on TB2 and compare with alternative NLP frameworks (e.g., FLAIR, spaCy) and transformer-based models. The Stanza tokenizer and lemmatizer achieve SOTA performance on TB2, while the Stanza NER tagger, part-of-speech (POS) tagger, and dependency parser achieve competitive performance against non-transformer models. The transformer-based models establish a strong baseline in Tweebank-NER and achieve the new SOTA performance in POS tagging and dependency parsing on TB2. We release the dataset and make both the Stanza pipeline and BERTweet-based models available “off-the-shelf” for use in future Tweet NLP research. Our source code, data, and pre-trained models are available at: https://github.com/social-machines/TweebankNLP. Keywords: text annotation, noisy text, NLP toolkit, Twitter, named entity recognition, tokenization, lemmatization, part-of-speech tagging, dependency parsing 1. Introduction Researchers use text data from social media platforms such as Twitter and Reddit for a wide range of studies including opinion mining, socio-cultural analysis, and language variation. Messages posted to such platforms are typically written in a less formal style than what are found in conventional data sources for NLP mod- els, namely news articles, papers, websites, and books. Processing the noisy and informal language of social media is challenging for traditional NLP tools because such messages are usually short in length and irreg- ular in spelling and structure. In response, the NLP community has been constructing language resources and building NLP pipelines for social media data, es- pecially for Twitter. Annotating social media language resources is impor- tant to the development of NLP tools. Foster et al. (2011) is the one of the earliest attempts to anno- tate tweets in the Penn Treebank (PTB) format. Fol- lowing a similar PTB-style convention suggested by Schneider et al. (2013), Kong et al. (2014) cre- ated Tweebank V1. However, the PTB annotation guidelines leave many annotation decisions unspeci- fied and are therefore …

Content truncated to 500 words