Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3819–3834 July 9-14, 2023 ©2023 Association for Computational Linguistics Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization Ze-Feng Gao1 , Kun Zhou2,3 , Peiyu Liu1,3 , Wayne Xin Zhao1,3∗and Ji-Rong Wen1,2,3 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3Beijing Key Laboratory of Big Data Management and Analysis Methods {zfgao,francis_kun_zhou,liupeiyustu,jrwen}@ruc.edu.cn, batmanfly@gmail.com Abstract By scaling the model size, large pre-trained language models (PLMs) have shown remark- able performance in various natural language processing tasks, mostly outperforming small PLMs by a large margin. However, due to the high computational cost, the huge number of parameters also restricts the applicability of large PLMs in real-world systems. In this paper, we focus on scaling up the parameters of PLMs only during fine-tuning, to benefit from the over-parameterization, while without increas- ing the inference latency. Given a relatively small PLM, we over-parameterize it by employ- ing a matrix product operator, an efficient and almost lossless decomposition method to factor- ize its contained parameter matrices into a set of higher-dimensional tensors. Considering the efficiency, we further propose both static and dynamic strategies to select the most important parameter matrices for over-parameterization. Extensive experiments have demonstrated that our approach can significantly boost the fine- tuning performance of small PLMs and even help small PLMs outperform 3× parameterized larger ones. Our code is publicly available at https://github.com/zfgao66/OPF. 1 Introduction Due to the remarkable performance, fine- tuning large-scale pre-trained language mod- els (PLMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Zhao et al., 2023) has become the de facto method in the natural language pro- cess (NLP) field. With the help of large-scale pre-trained data and parameters, these large-scale PLMs are able to process an extensive range of world knowledge (Roberts et al., 2020; Jiang et al., 2020) and generalize well on a variety of tasks (Brown et al., 2020; Lester et al., 2021). Fol- lowing this direction, more data and more param- eters have been widely explored in improving the ∗Corresponding author. model capacity of PLMs in recent years (Raffel et al., 2020; Chowdhery et al., 2022; Chen et al., 2022), leading to the expansion of PLM parameters from millions to billions. Despite the effectiveness, the huge model size also limits the use of large PLMs in both research and industry communities. Specifically, the com- putational cost of pre-training and the efficiency of utilizing large PLMs are often unaffordable for researchers as well as resource-limited appli- cations. Therefore, as a compromise, a number of studies (Gururangan et al., 2020; Chang et al., 2020; Zhang et al., 2020) focus on pre-training rel- atively smaller language models (e.g., BERT-base- uncased) on the domain-specific or task-specific corpus. However, small PLMs are not highly over-parameterized as larger models. As a re- sult, their generalization capabilities are relatively weaker (Brown et al., 2020), leading to a sub- optimal fine-tuning …

Content truncated to 500 words