Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5388–5398 August 1–6, 2021. ©2021 Association for Computational Linguistics 5388 Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators Peiyu Liu1,4∗, Ze-Feng Gao2,1∗ , Wayne Xin Zhao1,4,5† , Z.Y. Xie2, Zhong-Yi Lu2† and Ji-Rong Wen1,3,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2Department of Physics, Renmin University of China 3 School of Information, Renmin University of China 4Beijing Key Laboratory of Big Data Management and Analysis Methods 5Beijing Academy of Artificial Intelligence, Beijing, 100084, China {liupeiyustu,zfgao,qingtaoxie,zlu,jrwen}@ruc.edu.cn, batmanfly@gmail.com Abstract This paper presents a novel pre-trained lan- guage models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics. It can decompose an original matrix into cen- tral tensors (containing the core information) and auxiliary tensors (with only a small pro- portion of parameters). With the decomposed MPO structure, we propose a novel fine-tuning strategy by only updating the parameters from the auxiliary tensors, and design an optimiza- tion algorithm for MPO-based approximation over stacked network architectures. Our ap- proach can be applied to the original or the compressed PLMs in a general way, which derives a lighter network and significantly re- duces the parameters to be fine-tuned. Exten- sive experiments have demonstrated the effec- tiveness of the proposed approach in model compression, especially the reduction in fine- tuning parameters (91% reduction on average). The code to reproduce the results of this pa- per can be found at https://github.com/ RUCAIBox/MPOP. 1 Introduction Recently, pre-trained language models (PLMs) (De- vlin et al., 2019; Peters et al., 2018; Radford et al., 2018) have made significant progress in various natural language processing tasks. Instead of train- ing a model from scratch, one can fine-tune a PLM to solve some specific task through the paradigm of “pre-training and fine-tuning”. Typically, PLMs are constructed with stacked Transformer layers (Vaswani et al., 2017), involv- ing a huge number of parameters to be learned. Though effective, the large model size makes it im- practical for resource-limited devices. Therefore, there is an increasing number of studies focused ∗Authors contributed equally. †Corresponding author. on the parameter reduction or memory reduction of PLMs (Noach and Goldberg, 2020), including parameter sharing (Lan et al., 2020), knowledge distillation (Sanh et al., 2019), low-rank approxima- tion (Ma et al., 2019) and data quantization (Hubara et al., 2017). However, these studies mainly apply parameter reduction techniques to PLM compres- sion, which may not be intrinsically appropriate for the learning paradigm and architecture of PLMs. The compressed parameters are highly coupled so that it is difficult to directly manipulate different parts with specific strategies. For example, most PLM compression methods need to fine-tune the whole network architecture, although only a small proportion of parameters will significantly change during fine-tuning (Liu et al., 2020). In this paper, we introduce a novel matrix prod- uct operator (MPO) technique from quantum many- body physics for …

Content truncated to 500 words