DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8208–8222 July 9-14, 2023 ©2023 Association for Computational Linguistics DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models Xuxi Chen1, Tianlong Chen1, Weizhu Chen2, Ahmed Hassan Awadallah2 Zhangyang Wang1, Yu Cheng2 1University of Texas at Austin, 2Microsoft Corporation {xxchen,tianlong.chen,atlaswang}@utexas.edu {wzchen,hassanam,yu.cheng}@microsoft.com Abstract Gigantic pre-trained models have become cen- tral to natural language processing (NLP), serv- ing as the starting point for fine-tuning to- wards a range of downstream tasks. How- ever, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., 175B parameters for GPT-3), even the fine-tuning process can be time-consuming and computationally expensive; (b) the fine- tuned model has the same size as its start- ing point by default, which is neither sensible due to its more specialized functionality, nor practical since many fine-tuned models will be deployed in resource-constrained environ- ments. To address these pain points, we pro- pose a framework for resource- and parameter- efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning - by enforc- ing sparsity-aware low-rank updates on top of the pre-trained weights; and (ii) resource- efficient inference - by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two di- rections by exploiting both unstructured and structured sparse patterns in pre-trained lan- guage models via a unified approach. Exten- sive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, RoBERTa, and GPT-2) on dozens of datasets, consistently demonstrate impressive parameter- /inference-efficiency, while maintaining com- petitive downstream performance. For instance, DSEE saves about 25% inference FLOPs while achieving comparable performance, with 0.5% trainable parameters on BERT. Codes are avail- able at https://github.com/VITA-Group/ DSEE. 1 Introduction Most recent NLP applications have been following the pre-train then fine-tune paradigm, starting from a gigantic pre-trained model and fine-tuning it to- wards downstream tasks. Conventional fine-tuning works through updating all of the parameters in the pre-trained model. However, as the size of pre-trained models grows, updating all parameters becomes less feasible in most practical scenarios, due to the expensive memory and computational requirements. For example, BERTBASE (Devlin et al., 2019) has 110M trainable parameters, while GPT-2 (Radford et al., 2019) has up to 1.5B and the largest version of GPT-3 (Radford et al., 2019) has an astonishing 175B trainable parameters. As such, conventional fine-tuning of the larger models could require hundreds of GPU hours. Another downside of this paradigm is that it requires stor- ing as many parameters as in the large-scale pre- trained models for each downstream task, which poses impediments to the deployment in real-world resource-constrained environments. One solution to address the extensive resource requirement of conventional fine-tuning is model pruning (LeCun et al., 1990; Han et al., 2015; Ren et al., 2018; He et al., 2017; Liu et al., 2017), …

Content truncated to 500 words