Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3813–3827 August 1–6, 2021. ©2021 Association for Computational Linguistics 3813 Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level Ruiqi Zhong Dhruba Ghosh Dan Klein Jacob Steinhardt Computer Science Division, University of California, Berkeley {ruiqi-zhong, djghosh13, klein, jsteinhardt}@berkeley.edu Abstract Larger language models have higher accu- racy on average, but are they better on ev- ery single instance (datapoint)? Some work suggests larger models have higher out-of- distribution robustness, while other work sug- gests they have lower accuracy on rare sub- groups. To understand these differences, we investigate these models at the level of indi- vidual instances. However, one major chal- lenge is that individual predictions are highly sensitive to noise in the randomness in train- ing. We develop statistically rigorous meth- ods to address this, and after accounting for pretraining and finetuning noise, we find that our BERT-LARGE is worse than BERT-MINI on at least 1−4% of instances across MNLI, SST-2, and QQP, compared to the overall ac- curacy improvement of 2−10%. We also find that finetuning noise increases with model size, and that instance-level accuracy has mo- mentum: improvement from BERT-MINI to BERT-MEDIUM correlates with improvement from BERT-MEDIUM to BERT-LARGE . Our findings suggest that instance-level predictions provide a rich source of information; we there- fore recommend that researchers supplement model weights with model predictions. 1 Introduction Historically, large deep learning models (Peters et al., 2018; Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2019) have improved the state of the art on a wide range of tasks and leaderboards (Schwartz et al., 2014; Rajpurkar et al., 2016; Wang et al., 2018), and empirical scaling laws predict that larger models will continue to increase per- formance (Kaplan et al., 2020). However, little is understood about such improvement at the instance (datapoint) level. Are larger models uniformly bet- ter? In other words, are larger pretrained models better at every instance, or are they better at some instances, but worse at others? Prior works hint at differing answers. Hendrycks et al. (2020) and Desai and Durrett (2020) find that larger pretrained models consistently improve out-of-distribution performance, which implies that they might be uniformly better at a finer level. Henighan et al. (2020) claim that larger pretrained image models have lower downstream classifica- tion loss for the majority of instances, and they predict this trend to be true for other data modal- ities (e.g. text). On the other hand, Sagawa et al. (2020) find that larger non-pretrained models per- form worse on rare subgroups; if this result gener- alizes to pretrained language models, larger models will not be uniformly better. Despite all the in- direct evidence, it is still inconclusive how many instances larger pretrained models perform worse on. A na¨ıve solution is to finetune a larger model, compare it to a smaller one, and find instances where the larger model is worse. However, this approach is flawed, since model predictions are noisy at the …

Content truncated to 500 words