TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1644 TWEETEVAL: Unified Benchmark and Comparative Evaluation for Tweet Classification Francesco Barbieri♣ Jose Camacho-Collados† Leonardo Neves♣ Luis Espinosa-Anke† ♣Snap Inc., Santa Monica, CA 90405, USA †School of Computer Science and Informatics, Cardiff University, United Kingdom ♣{fbarbieri,lneves}@snap.com, †{camachocolladosj,espinosa-ankel}@cardiff.ac.uk Abstract The experimental landscape in natural lan- guage processing for social media is too frag- mented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain- specific data. In this paper, we propose a new evaluation framework (TWEETEVAL) consist- ing of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and com- pare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre- trained generic language models, and continue training them on Twitter corpora. 1 Introduction Modern NLP systems are typically ill-equipped when applied to noisy user-generated text. The high-paced, conversational and idiosyncratic na- ture of social media, paired with platform-specific restrictions (e.g., Twitter’s character limit), requires tackling additional challenges, for example, POS tagging (Derczynski et al., 2013), lexical normal- ization (Han and Baldwin, 2011; Baldwin et al., 2015), or named entity recognition (Ritter et al., 2011; Baldwin et al., 2013). In other more generic contexts, these challenges can be considered solved or are simply non-existent. Moreover, other ap- parently simple tasks such as sentiment analysis have proven to be hard on Twitter data (Poria et al., 2020), among others, due to limited amount of contextual cues available in short texts (Kim et al., 2014). In addition to these and other inherent dif- ficulties, advances in NLP for user-generated data are hindered by its highly fragmented landscape and the lack of a unified evaluation framework. In the current era of pretraining and Language Mod- els (LMs), this is particularly relevant, as these models exhibit a versatility that currently cannot be gauged comparably across Twitter datasets and tasks. This is not the case, however, in more or- dinary textual genres and domains. For instance, well known benchmarks like SentEval (Conneau and Kiela, 2018), GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) include standard NLP tasks such as language inference, paraphrase detection or sentiment analysis, among others. It is undisputable that these benchmarks have con- tributed to the fast development of language under- standing techniques, and LMs in particular, as they have enabled comprehensive evaluations across several tasks in fair and reproducible conditions. We thus take inspiration from the above to de- velop TWEETEVAL, a benchmark for tweet classi- fication in English. TWEETEVAL is a standardized test bed for seven tweet classification tasks. These are: sentiment analysis, emotion recognition, of- fensive language detection, hate speech detection, stance …

Content truncated to 500 words