Bernice: A Multilingual Pre-trained Encoder for Twitter

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6191 - 6205 December 7-11, 2022 ©2022 Association for Computational Linguistics Bernice: A Multilingual Pre-trained Encoder for Twitter Alexandra DeLucia, Shijie Wu∗, Aaron Mueller, Carlos Aguirre, and Mark Dredze Center for Language and Speech Processing Johns Hopkins University {aadelucia,shijie.wu,amueller,caguirr4,mdredze}@jhu.edu Philip Resnik Linguistics/UMIACS University of Maryland resnik@umd.edu Abstract The language of Twitter differs significantly from that of other domains commonly included in large language model training. While tweets are typically multilingual and contain informal language, including emoji and hashtags, most pre-trained language models for Twitter are ei- ther monolingual, adapted from other domains rather than trained exclusively on Twitter, or are trained on a limited amount of in-domain Twitter data. We introduce Bernice, the first multilingual RoBERTa language model trained from scratch on 2.5 billion tweets with a cus- tom tweet-focused tokenizer. We evaluate on a variety of monolingual and multilingual Twit- ter benchmarks, finding that our model consis- tently exceeds or matches the performance of a variety of models adapted to social media data as well as strong multilingual baselines, despite being trained on less data overall. We posit that it is more efficient compute- and data-wise to train completely on in-domain data with a specialized domain-specific tokenizer. https://github.com/JHU-CLSP/ Bernice-Twitter-encoder 1 Introduction BERT (Devlin et al., 2019) has become a core part of modern natural language processing (NLP) pipelines. Performance on downstream tasks de- pends on the quality of the included BERT-style model, driven in large part by model size, vocab- ulary, and relevance of pre-training data. Studies have shown that pre-training data should match the intended deployment domain to achieve optimal task performance (Barbieri et al., 2021; Nguyen et al., 2020; Huang et al., 2020a; Gururangan et al., 2020; Gu et al., 2021; Zhang et al., 2022). One of the most commonly studied domains for NLP research is Twitter, which has led to numer- ous downstream applications. The unique nature of ∗Currently at Bloomberg. Work performed while at Johns Hopkins. Multilingual Twitter pre-training only Twitter tokenizer BERTweet XLM-R XLM-T TwHIN-BERT* Bernice Table 1: A comparison of commonly used models for tweet representation in downstream tasks. Bernice is the first multilingual model trained from scratch on Twitter data with a custom tokenizer. Twitter data has prompted specialized applications, including sentiment prediction around elections and vaccines (Broniatowski et al., 2018), stance predic- tion on gun control (Benton and Dredze, 2018), civil unrest forecasting (Chinta et al., 2021; Sech et al., 2020; Alsaedi et al., 2017), and various men- tal health applications (Harrigian et al., 2021). While several different BERT-style models have been developed for Twitter (Barbieri et al., 2021, 2020; Nguyen et al., 2020; Zhang et al., 2022), com- pared to BERT models for other domains, these Twitter-based models are smaller, use much less training data, and are often adapted from other do- mains instead of for Twitter initially. This is es- pecially problematic for the tokenizer, creating a mismatch that limits its efficacy on representing Twitter data. …

Content truncated to 500 words