When Do You Need Billions of Words of Pretraining Data?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1112–1125 August 1–6, 2021. ©2021 Association for Computational Linguistics 1112 When Do You Need Billions of Words of Pretraining Data? Yian Zhang,∗,1 Alex Warstadt,∗,2 Haau-Sing Li,3 and Samuel R. Bowman1,2,3 1Dept. of Computer Science, 2Dept. of Linguistics, 3Center for Data Science New York University {yian.zhang, warstadt, xl3119, bowman}@nyu.edu Abstract NLP is currently dominated by language mod- els like RoBERTa which are pretrained on billions of words. But what exact knowl- edge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? To explore this question, we adopt five styles of evaluation: classifier probing, information-theoretic probing, unsu- pervised relative acceptability judgments, un- supervised language model knowledge prob- ing, and fine-tuning on NLU tasks. We then draw learning curves that track the growth of these different measures of model ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that these LMs require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features we test. They need a much larger quantity of data in order to acquire enough commonsense knowl- edge and other skills required to master typi- cal downstream NLU tasks. The results sug- gest that, while the ability to encode linguis- tic features is almost certainly necessary for language understanding, it is likely that other, unidentified, forms of knowledge are the ma- jor drivers of recent improvements in language understanding among large pretrained models. 1 Introduction Pretrained language models (LMs) like BERT and RoBERTa have become ubiquitous in NLP. New models require massive datasets of tens or even hundreds of billions of words (Brown et al., 2020) to improve on existing models on language un- derstanding benchmarks like GLUE (Wang et al., 2018). Much recent work has used probing meth- ods to evaluate what these models do and do not *Equal Contribution None 1M 10M 100M 1B 30B 0.0 0.2 0.4 0.6 0.8 1.0 Classifier Probing (Edge Probing) MDL Reflected (Edge Probing) BLiMP LAMA SuperGLUE Relative Performance Pretraining Dataset Size Figure 1: Overall learning curves for the five evaluation methods. For each method, we compute overall perfor- mance for each RoBERTa model tested as the macro av- erage over sub-task’s performance after normalization. We fit an exponential curve which we scale to have an initial value of 0 and an asymptote at 1. Classifier and MDL probing mainly test models’ encoding of lin- guistic features; BLiMP tests model’s understanding of linguistic phenomena; LAMA tests factual knowledge; SuperGLUE is a suite of conventional NLU tasks. learn (Belinkov and Glass, 2019; Tenney et al., 2019b; Rogers et al., 2020; Ettinger, 2020). Since most of these works only focus on models pre- trained on a fixed data volume (usually billions of words), many interesting questions regarding the effect of the amount of pretraining …

Content truncated to 500 words