PoliBERTweet: A Pre-trained Language Model for Analyzing Political Content on Twitter

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7360–7367 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7360 PoliBERTweet: A Pre-trained Language Model for Analyzing Political Content on Twitter Kornraphop Kawintiranon, Lisa Singh Department of Computer Science Georgetown University Washington, DC, USA {kk1155,lisa.singh}@georgetown.edu Abstract Transformer-based models have become the state-of-the-art for numerous natural language processing (NLP) tasks, especially for noisy data sets, including social media posts. For example, BERTweet, pre-trained RoBERTa on a large amount of Twitter data, has achieved state-of-the-art results on several Twitter NLP tasks. We argue that it is not only important to have general pre-trained models for a social media platform, but also domain-specific ones that better capture domain-specific language context. Domain-specific resources are not only important for NLP tasks associated with a specific domain, but they are also useful for understanding language differences across domains. One domain that receives a large amount of attention is politics, more specifically political elections. Towards that end, we release PoliBERTweet, a pre-trained language model trained from BERTweet on over 83M US 2020 election-related English tweets. While the construction of the resource is fairly straightforward, we believe that it can be used for many important downstream tasks involving language, including political misinformation analysis and election public opinion analysis. To show the value of this resource, we evaluate PoliBERTweet on different NLP tasks. The results show that our model outperforms general-purpose language models in domain-specific contexts, highlighting the value of domain-specific models for more detailed linguistic analysis. We also extend other existing language models with a sample of these data and show their value for presidential candidate stance detection, a context-specific task. We release PoliBERTweet and these other models to the community to advance interdisciplinary research related to Election 2020. Keywords: Transformer, BERT, RoBERTa, BERTweet, Language Model, Politics, Election 2020, Twitter 1. Introduction Transformer-based language models (LM) have be- come the state-of-the-art algorithms for many NLP tasks. While the most well-known transformers are BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), the number of variants continue to grow (Jiang et al., 2020b; Joshi et al., 2019; Sanh et al., 2019; Lan et al., 2020). In principle, each model utilizes the unlabeled text corpora using different objective learn- ing functions, for example, masked language model- ing (MLM), next sentence prediction (NSP), or sen- tence order prediction (SOP) (Vaswani et al., 2017). All these mechanisms and functions have the same goal – to learn semantic representation with respect to the training corpus and generate contextualized embed- dings given a textual input. Recently, BERTweet, a pre-trained RoBERTa model trained on Twitter data collected from 2012 to 2019 was released (Nguyen et al., 2020). While this lan- guage model is valuable as a general-purpose Twitter- based model, we believe that data scientists and com- putational social scientists will find it valuable to have a model that is focused on recent political and election semantics related to the 2020 US presidential election. We …

Content truncated to 500 words