Downstream Datasets Make Surprisingly Good Pretraining Corpora

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12207–12222 July 9-14, 2023 ©2023 Association for Computational Linguistics Downstream Datasets Make Surprisingly Good Pretraining Corpora Kundan Krishna, Saurabh Garg, Jeffrey P. Bigham, Zachary C. Lipton Carnegie Mellon University {kundank,sgarg2,jbigham,zlipton}@andrew.cmu.edu Abstract For most natural language processing tasks, the dominant practice is to finetune large pretrained transformer models (e.g., BERT) using smaller downstream datasets. Despite the success of this approach, it remains unclear to what ex- tent these gains are attributable to the massive background corpora employed for pretraining versus to the pretraining objectives themselves. This paper introduces a large-scale study of self-pretraining, where the same (downstream) training data is used for both pretraining and finetuning. In experiments addressing both ELECTRA and RoBERTa models and 10 dis- tinct downstream classification datasets, we ob- serve that self-pretraining rivals standard pre- training on the BookWiki corpus (despite using around 10×–500× less data), outperforming the latter on 7 and 5 datasets, respectively. Sur- prisingly, these task-specific pretrained models often perform well on other tasks, including the GLUE benchmark. Self-pretraining also pro- vides benefits on structured output prediction tasks such as question answering and common- sense inference, often providing more than 50% improvements compared to standard pretrain- ing. Our results hint that often performance gains attributable to pretraining are driven pri- marily by the pretraining objective itself and are not always attributable to the use of exter- nal pretraining data in massive amounts. These findings are especially relevant in light of con- cerns about intellectual property and offensive content in web-scale pretraining data.1 1 Introduction For training predictive models operating on natural language data, the current best practice is to pre- train models on large unlabeled upstream corpora to optimize self-supervised objectives, for exam- ple, masked language modeling (MLM); the result- ing weights are then used to initialize models that 1Pretrained models can be downloaded from https:// github.com/acmi-lab/self-pretrain are subsequently trained (finetuned) on the labeled downstream data available for the task at hand. Large-scale pretrained models typically provide significant performance boosts when compared to models trained directly on the downstream task (with random initializations) (Peters et al., 2018; Devlin et al., 2019; Chiang and Lee, 2020; Kr- ishna et al., 2021). Upstream corpora tend to be significantly larger than the downstream corpora and the success of this approach is often attributed to its ability to leverage these massive upstream corpora (Liu et al., 2019; Yang et al., 2019). For example, the seminal BERT model (Devlin et al., 2019) was pretrained using the BookWiki corpus which is a combination of English Wikipedia and BooksCorpus (Zhu et al., 2015), totaling 13GB of plain text. Subsequent models have moved on to web-scale data. For example, XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), and T5 (Raffel et al., 2020)), were trained on 158GB, 160GB and 750GB of data, respectively. As upstream corpus size and downstream per- formance have gone up, popular attempts at ex- plaining these gains have focused …

Content truncated to 500 words