Benchmarking of Transformer-Based Pre-Trained Models on Social Media Text Classification Datasets

Benchmarking of Transformer-Based Pre-Trained Models on Social Media Text Classification Datasets Yuting Guo*1, Xiangjue Dong*1, Mohammed Ali Al-Garadi2, Abeed Sarker2, Cécile Paris3, Diego Mollá-Aliod4 1Department of Computer Science, Emory University, Atlanta, GA, USA 2Department of Biomedical Informatics, Emory University, Atlanta, GA, USA 3CSIRO Data61, Sydney, Australia 4Department of Computing, Macquarie University, Sydney, Australia {yuting.guo, xiangjue.dong, m.a.al-garadi, abeed.sarker}@emory.edu cecile.paris@data61.csiro.au, diego.molla-aliod@mq.edu.au Abstract Free text data from social media is now widely used in natural language processing research, and one of the most common ma- chine learning tasks performed on this data is classification. Generally speaking, per- formances of supervised classification algo- rithms on social media datasets are lower than those on texts from other sources, but recently- proposed transformer-based models have con- siderably improved upon legacy state-of-the- art systems. Currently, there is no study that compares the performances of different variants of transformer-based models on a wide range of social media text classification datasets. In this paper, we benchmark the performances of transformer-based pre-trained models on 25 social media text classifica- tion datasets, 6 of which are health-related. We compare three pre-trained language mod- els, RoBERTa-base, BERTweet and Clinical- BioBERT in terms of classification accuracy. Our experiments show that RoBERTa-base and BERTweet perform comparably on most datasets, and considerably better than Clinical- BioBERT, even on health-related datasets. 1 Introduction Transformer-based pre-trained language models have proven to be effective for many natural lan- guage processing (NLP) tasks, such as text clas- sification and question answering, and they have enabled systems to outperform previous state-of- the-art approaches. A prime example of such lan- guage representation models is Bidirectional En- coder Representations from Transformers (BERT), which was pre-trained on the Book Corpus and En- glish Wikipedia (Devlin et al., 2019). Since it was proposed, many efforts have attempted to improve upon it, and common strategies for doing so are to use more data and train longer (Liu et al., 2019), or to pre-train from scratch on domain-specific data (Gu et al., 2020). Multiple variants of transformer- based models have been proposed, but there is cur- rently limited information available about how the variants directly compare on a set of similar tasks. In this paper, we focus on text from a specific source, namely, social media, and the common task of text classification. We compare the perfor- mances of three pre-training methods. We chose text classification as our target task because it is perhaps the most common NLP-related machine learning task, and most of the publicly-available an- notated datasets were prepared for it. We included 25 social media classification datasets, 6 of which are health-related. We compared three transformer- based models—RoBERTa-base (Liu et al., 2019), BERTweet (Nguyen et al., 2020a), and ClinicalBio- BERT (Alsentzer et al., 2019). Our experiments show that RoBERTa-base and BERTweet perform comparably and are considerably better than Clin- icalBioBERT. In addition to comparing the per- formances of the models on all the datasets, we analyzed the differences in performances between domain-specific (medical), source-specific (social media), and generic pre-trained models. Our empir- ical …

Content truncated to 500 words