Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7319–7328 August 1–6, 2021. ©2021 Association for Computational Linguistics 7319 Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning Armen Aghajanyan Facebook AI armenag@fb.com Sonal Gupta Facebook sonalgupta@fb.com Luke Zettlemoyer Facebook AI University of Washington lsz@fb.com Abstract Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understand- ing tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gra- dient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine- tuning through the lens of intrinsic dimension provides us with empirical and theoretical intu- itions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; there exists a low dimension reparameteriza- tion that is as effective for fine-tuning as the full parameter space. For example, by optimiz- ing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre- training implicitly minimizes intrinsic dimen- sion and, perhaps surprisingly, larger mod- els tend to have lower intrinsic dimension af- ter a fixed number of pre-training updates, at least in part explaining their extreme effective- ness. Lastly, we connect intrinsic dimensional- ity with low dimensional task representations and compression based generalization bounds to provide generalization bounds that are inde- pendent of the full parameter count. 1 Introduction Pre-trained language models (Radford et al., 2019; Devlin et al., 2018; Liu et al., 2019; Lewis et al., 2019, 2020) provide the defacto initialization for modeling most existing NLP tasks. However, the process of fine-tuning them on often very small target task datasets remains somewhat mysterious. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of param- eters on datasets with only hundreds or thousands of labeled examples? We propose intrinsic dimensionality as a new lens through which fine-tuning can be analyzed (Li et al., 2018). An objective function’s intrinsic dimensionality describes the minimum dimension needed to solve the optimization problem it de- fines to some precision level. In the context of pre-trained language models, measuring intrinsic dimensional will tell us how many free parameters are required to closely approximate the optimiza- tion problem that is solved while fine-tuning for each end task. For example, we will show that 200 parameters (randomly projected back into the full parameter space) are enough to represent the prob- lem of tuning a RoBERTa model to within 90% of the performance of the full model. More gen- erally, we also describe a set of strong empirical and theoretical …

Content truncated to 500 words