Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1701–1713 July 9-14, 2023 ©2023 Association for Computational Linguistics Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models Zhong Zhang1,2, Bang Liu3,∗,†, Junming Shao1,2,† 1University of Electronic Science and Technology of China, Chengdu, China 2Shenzhen Institute for Advanced Study, UESTC, Shenzhen, China 3Mila & Université de Montréal, Montréal, Canada zhongzhang@std.uestc.edu.cn, bang.liu@umontreal.ca, junmshao@uestc.edu.cn Abstract Pre-trained language models (PLMs) are known to be overly parameterized and have significant redundancy, indicating a small de- gree of freedom of the PLMs. Motivated by the observation, in this paper, we study the problem of re-parameterizing and fine-tuning PLMs from a new perspective: Discovery of intrinsic task-specific subspace. Specifically, by exploiting the dynamics of the fine-tuning process for a given task, the parameter opti- mization trajectory is learned to uncover its in- trinsic task-specific subspace. A key finding is that PLMs can be effectively fine-tuned in the subspace with a small number of free parame- ters. Beyond, we observe some outlier dimen- sions emerging during fine-tuning in the sub- space. Disabling these dimensions degrades the model performance significantly. This sug- gests that these dimensions are crucial to in- duce task-specific knowledge to downstream tasks. 1 Introduction Pre-trained Language Models (PLMs) have be- come the de facto methods for various natural language processing (NLP) tasks (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019). The typical paradigm is to pre-train a big language model on large-scale corpora and then fine-tune the model on small task-specific datasets to adapt to the downstream tasks. Despite the great success of this paradigm, two questions still come to our mind: (1) Why can a PLM with hundreds of mil- lions of parameters be successfully fine-tuned on different downstream tasks using only hundreds or thousands of labeled samples? (2) Do we re- ally need a full fine-tuning of all parameters of a PLM to reach state-of-the-art performance on downstream tasks? In this paper, we try to pro- vide a new viewpoint on the two questions, and ∗ Canada CIFAR AI Chair. † Corresponding authors. claim that: Fine-tuning happens only in some tiny task-specific subspaces, which can be effec- tively learned with a small number of free pa- rameters. Recent studies have shown that PLMs are highly over-parameterized and robust to pruning (Frankle and Carbin, 2019; Chen et al., 2020; Prasanna et al., 2020; Gordon et al., 2020; Liang et al., 2021, 2022), and can be fine-tuned in parameter-efficient ways (Gong et al., 2022; Za- ken et al., 2022; Mahabadi et al., 2021; Li and Liang, 2021). This emerging empirical evidence tends to point to one fact that there exist some in- trinsic structures in PLMs that are responsible for inducing task-specific knowledge to downstream tasks. Notably, the recent work (Aghajanyan et al., 2021) provides a promising conclusion that PLMs can be re-parameterized and fine-tuned in random low-dimensional subspaces using random projec- tion, and the dimensionality of the random …

Content truncated to 500 words