To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2209–2213 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2209 To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks Sinong Wang, Madian Khabsa, Hao Ma Faceboook AI, Seattle, USA {sinongwang, mkhabsa, haom}@fb.com Abstract Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the ben- efits of pretrained models as a function of the number of training samples used in the downstream task. On several text classifica- tion tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a dimin- ishing return point as the supervised data size increases significantly. 1 Introduction Language modeling has emerged as an effective pretraining approach in wide variety of NLP mod- els. Multiple techniques have been proposed, in- cluding bi-directional language modeling (Peters et al., 2018), masked language models (Devlin et al., 2018), and variants of denoising auto-encoder approaches (Lewis et al., 2019; Raffel et al., 2019; Joshi et al., 2019). Today, it is rare to examine a leaderboard without finding the top spots occupied by some variant of a pretraining method.1 The future of NLP appears to be paved by pre- training a universal contextual representation on wikipedia-like data at massive scale. Attempts along this path have pushed the frontier to up 10× to the size of wikipedia (Raffel et al., 2019). How- ever, the success of these experiments is mixed: although improvements have been observed, the downstream task is usually data-limited. There is evidence that large-scale pretraining does not al- ways lead to state-of-the-art results (Raffel et al., 2019), especially on tasks such as machine trans- lation, where abundance of training data, and the 1https://super.gluebenchmark.com/leaderboard existence of strong augmentation methods such as back translation might have limited the benefit of pretraining. This paper examines the pretraining benefits of downstream tasks as the number of training sam- ples increases. To answer this question, we fo- cus on multi-class text classification since: (i) it is one of most important problems in NLP with applications spanning multiple domains. (ii) large sums of training data exists for many text classifi- cation tasks, or can be obtained relatively cheaply through crowd workers (Snow et al., 2008). We choose three sentiment classification datasets: Yelp review (yel, 2019), Amazon sports and electronics review (Ni et al., 2019), ranging in size from 6 to 18 million examples. 2 We finetune a RoBERTa model (Liu et al., 2019) with increments of the downstream dataset, and evaluate the performance at each increment. For example, on the Yelp dataset whose size is 6 mil- lion, we train the models on subsets of the data with each subset size being in the sequence (60k, 600K, 1.8M, 3M .., 6M). For …

Content truncated to 500 words