Numerical Optimizations for Weighted Low-rank Estimation on Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1404 - 1416 December 7-11, 2022 ©2022 Association for Computational Linguistics Numerical Optimizations for Weighted Low-rank Estimation on Language Model Ting Hua∗, Yen-Chang Hsu∗, Felicity Wang, Qian Lou, Yilin Shen, Hongxia Jin Samsung Research America {ting.hua,yenchang.hsu,f.wang1,qian.lou}@samsung.com {yilin.shen,hongxia.jin}@samsung.com Abstract Singular value decomposition (SVD) is one of the most popular compression methods that ap- proximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The pa- rameters of a trained neural network model may affect the task performance unevenly, which suggests non-equal importance among the pa- rameters. Compared to SVD, the decomposi- tion method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighted value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically in- vestigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models. Further, we designed a metric to pre- dict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy. The extensive evaluations demonstrate that our method can perform bet- ter than current SOTA methods in compressing Transformer-based language models. 1 Introduction Transformer-based language models such as BERT (Devlin et al., 2018) have obtained significant suc- cess in a variety of Natural Language Processing tasks, such as language modeling (Radford et al., 2018), text classification (Wang et al., 2018), ques- tion answering (Rajpurkar et al., 2016), and sum- marization (Liu, 2019). Despite their success, these models usually contain millions or even billions of parameters, pre-trained by the large corpus. How- ever, the downstream tasks may only focus on a specific scenario, such that only a small amount of parameters in the big Transformer model will con- tribute to the performance of the target task. Also, the massive size of Transformer models prohibits *These authors contributed equally to this work. their deployments to resource-constrained devices. Therefore, compression of the Transformer-based language model attracts extensive interests. Low-rank factorization (Golub and Reinsch, 1971; Noach and Goldberg, 2020) aims to approx- imate each parameter matrix in the trained model by two smaller matrices. This line of compression strategy will naturally inherit the knowledge of the big trained model without expensive generic re- training, and is the orthogonal direction to other compression approaches such as Knowledge dis- tillation (Sun et al., 2019; Sanh et al., 2019; Jiao et al., 2019) or Quantization (Shen et al., 2020; Zhao et al., 2021). However, applying standard SVD to approxi- mate the learned weights often results in a signifi- cant task performance drop. Previous work shows that this phenomenon may be caused by a strong assumption held by the standard SVD, that the pa- rameters in the matrix are equally crucial to the performance (Hsu et al., 2021). Also, it has been observed that different parameters in Transformer models have different impacts on the …

Content truncated to 500 words