Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1675–1681 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1675 Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media Xiang Dai1,2 Sarvnaz Karimi1 Ben Hachey3 Cecile Paris1 1CSIRO Data61, Sydney, Australia 2University of Sydney, Sydney, Australia 3Harrison ai, Sydney, Australia {dai.dai,sarvnaz.karimi,cecile.paris}@csiro.au ben.hachey@gmail.com Abstract Recent studies on domain-specific BERT mod- els show that effectiveness on downstream tasks can be improved when models are pre- trained on in-domain data. Often, the pre- training data used in these models are se- lected based on their subject matter, e.g., bi- ology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0. 1 Introduction Sequence transfer learning (Ruder, 2019), that pre- trains language representations on unlabeled text (source) and then adapts these representations to a supervised task (target), has demonstrated its effectiveness on a range of NLP tasks (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019). Approaches vary in model, pretraining objective, pretraining data and adaptation strategy. We con- sider a widely used method, BERT (Devlin et al., 2019). It pretrains a transformer-based model us- ing a masked language model objective and then fine-tunes the model on the target task. We inves- tigate the impact of the domain (i.e., the similar- ity between the underlying distribution of source and target data) of pretraining data on the effec- tiveness of pretrained models. We also propose a cost-effective way to select pretraining data. Recent studies on domain-specific BERT mod- els, which are pretrained on specialty source data, empirically show that, when in-domain data is used for pretraining, target task performance can be im- proved (Lee et al., 2019; Alsentzer et al., 2019; Huang et al., 2019; Beltagy et al., 2019). These publicly available domain-specific BERT models are valuable to the NLP community. However, the selection of in-domain data usually resorts to intu- ition, which varies across NLP practitioners (Dai et al., 2019). According to Halliday and Hasan (1989), the context specific usage of language is affected by three factors: field (the subject matter being discussed), tenor (the relationship between the participants in the discourse and their purpose) and mode (communication medium, e.g., ‘spoken’ or ‘written’).1 Generally, the selection of pretrain- ing data in existing domain-specific BERT models is based on the field rather than the tenor. For example, BioBERT (Lee et al., 2019) and SciB- ERT (Beltagy et al., 2019) are both pretrained on scholar articles, but on different fields (biology and computer science). We conduct a case study of pretraining BERT on social media text which has very different tenor from existing domain-specific BERT models. Our contributions are two-fold: (1) We release two pre- trained BERT …

Content truncated to 500 words