Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2376–2382 June 6–11, 2021. ©2021 Association for Computational Linguistics 2376 Rethinking Network Pruning— under the Pre-train and Fine-tune Paradigm Dongkuan Xu1 and Ian En-Hsu Yen2 and Jinxi Zhao2 and Zhibin Xiao2 1The Pennsylvania State University, State College, PA, USA 2Moffett AI, Los Altos, CA, USA {dux19}@psu.edu, {ian.yan, jx.zhao, zb.xiao}@moffett.ai Abstract Transformer-based pre-trained language mod- els have significantly improved the perfor- mance of various natural language processing (NLP) tasks in the recent years. While effec- tive and prevalent, these models are usually prohibitively large for resource-limited deploy- ment scenarios. A thread of research has thus been working on applying network pruning techniques under the pretrain-then-finetune paradigm widely adopted in NLP. However, the existing pruning results on benchmark transformers, such as BERT, are not as re- markable as the pruning results in the litera- ture of convolutional neural networks (CNNs). In particular, common wisdom in pruning CNN states that sparse pruning technique com- presses a model more than that obtained by re- ducing number of channels and layers (Elsen et al., 2020; Zhu and Gupta, 2017), while ex- isting works on sparse pruning of BERT yields inferior results than its small-dense counter- parts such as TinyBERT (Jiao et al., 2020). In this work, we aim to fill this gap by study- ing how knowledge are transferred and lost during the pre-train, fine-tune, and pruning process, and proposing a knowledge-aware sparse pruning process that achieves signifi- cantly superior results than existing literature. We show for the first time that sparse prun- ing compresses a BERT model significantly more than reducing its number of channels and layers. Experiments on multiple data sets of GLUE benchmark show that our method outperforms the leading competitors with a 20-times weight/FLOPs compression and ne- glectable loss in prediction accuracy1. 1 Introduction Pre-trained language models, such as BERT (De- vlin et al., 2019), become the standard and effective methods for improving the performance of a variety 1Codes can be found in the authors’ website. Work done during an internship at Moffett AI. of natural language processing (NLP) tasks. These models are pre-trained in a self-supervised fash- ion and then fine-tuned for supervised downstream tasks. However, these models suffer from the heavy model size, making them impractical for resource- limited deployment scenarios and incurring cost concerns (Strubell et al., 2019). In parallel, an emerging subfield has studied the redundancy in deep neural network models (Zhu and Gupta, 2017; Gale et al., 2019) and proposed to prune networks without sacrificing performance, such as the lottery ticket hypothesis (Frankle and Carbin, 2019). Common wisdom in CNN literature shows that sparse pruning leads to more compres- sion rate than structural pruning. For example, for the same number of parameters (0.46M), the sparse MobileNets improve by 11.2% accuracy over the dense ones (Zhu and Gupta, 2017). However, sim- ilar conclusions are not observed for pre-trained language models. The main question this paper attempts to …

Content truncated to 500 words