Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification

Proceedings of the 28th International Conference on Computational Linguistics, pages 5522–5529 Barcelona, Spain (Online), December 8-13, 2020 5522 Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification Aleksandra Edwards† Jose Camacho-Collados† H´el`ene de Ribaupierre† Alun Preece‡ †School of Computer Science and Informatics, Cardiff University, United Kingdom ‡Crime and Security Research Institute, Cardiff University, United Kingdom {edwardsai,camachocolladosj,deribaupierreh,preecead}@cardiff.ac.uk Abstract Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. However, most classification datasets assume a large amount labeled data, which is commonly not the case in practical settings. In particular, in this paper we compare the performance of a light-weight lin- ear classifier based on word embeddings, i.e., fastText (Joulin et al., 2017), versus a pre-trained language model, i.e., BERT (Devlin et al., 2019), across a wide range of datasets and classifica- tion tasks. In general, results show the importance of domain-specific unlabeled data, both in the form of word embeddings or language models. As for the comparison, BERT outperforms all baselines in standard datasets with large training sets. However, in settings with small training datasets a simple method like fastText coupled with domain-specific word embeddings performs equally well or better than BERT, even when pre-trained on domain-specific data. 1 Introduction Language models pre-trained on large amounts of text corpora form the foundation of today’s NLP (Gururangan et al., 2020; Rogers et al., 2020). They have proved to provide state-of-the-art performance against most standard NLP benchmarks (Wang et al., 2019a; Wang et al., 2019b). However, these models require large computational resources that are not always available and have important environment implications (Strubell et al., 2019). Moreover, there is limited research in the applicability of pre-trained models in classification tasks with small amount of labelled data. Some related studies (Lee et al., 2020; Nguyen et al., 2020; Huang et al., 2019; Alsentzer et al., 2019) investigate whether it is helpful to tailor a pre-trained model to the domain while others (Sun et al., 2019; Chronopoulou et al., 2019; Radford et al., 2018) analyse methods for fine-tuning BERT to a given task. However, these studies perform evaluation on a limited range of datasets and classification models and do not consider scenarios with limited amounts of training data. In particular, this paper aims to estimate the role of labeled and unlabeled data for supervised text classification. Our study is similar to Gururangan et al. (2020) where they investigate whether it is still helpful to tailor a pre-trained model to the domain of a target task. In this paper, however, we focus our evaluation on text classification and compare different types of classifiers on different domains (social media, news and reviews). Unlike other tasks such as natural language inference or question answering that may require a subtle understanding, feature-based linear models are still considered to be competitive in text classification (Kowsari et al., 2019). However, to the best of our knowledge there has not been an extensive comparison …

Content truncated to 500 words