Structured Pruning of Large Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6151–6162, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6151 Structured Pruning of Large Language Models Ziheng Wang * ASAPP, Inc. zihengw@stanford.edu Jeremy Wohlwend * ASAPP, Inc. jwohlwend@csail.mit.edu Tao Lei * ASAPP, Inc. tao@asapp.com Abstract Large language models have recently achieved state of the art performance across a wide va- riety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their us- age costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, struc- tured pruning approach by parameterizing each weight matrix using its low-rank factor- ization, and adaptively removing rank-1 com- ponents during training. On language model- ing tasks, our structured approach outperforms other unstructured and block-structured prun- ing baselines at various compression levels, while achieving significant speedups during both training and inference. We also demon- strate that our method can be applied to prun- ing adaptive word embeddings in large lan- guage models, and to pruning the BERT model on several downstream fine-tuning classifica- tion benchmarks.1 1 Introduction Recent advances in language modeling have led to remarkable improvements on a variety of natu- ral language tasks (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Dai et al., 2019; Zhang et al., 2019). These models, however, have grown increasingly large, rendering them slow and expensive for real- world applications. Through the use of model com- pression, we aim to reduce this overhead, and to better understand the role of model capacity in large language models. A common approach to model compression is known as weight pruning (Zhu and Gupta, 2017; Han et al., 2015a; See et al., 2016). Model weights *Denotes equal contribution. 1Our code is publicly available at https://github. com/asappresearch/flop. are progressively removed, resulting in sparse ma- trices across the network. Earlier work focuses mostly on unstructured pruning, where weights are pruned individually (Narang et al., 2017a; Zhu and Gupta, 2017). While this method is effective, it results in unstructured sparse matrices that are dif- ficult to support on common hardware (Han et al., 2016), making it challenging to obtain training and inference speedups despite a significant reduction in model size. On the other hand, structured pruning imposes structured sparse patterns by removing groups of consecutive parameters, such as rows, columns or k×k sub-blocks of the weight matrix (Narang et al., 2017b; Wen et al., 2018; Cao et al., 2019). These methods lead to significant speedup, but tend to give lower performance than unstructured pruning given the same parameter budget (Yao et al., 2019). Another caveat is that some of these methods re- quire special linear algebra implementations (Gray et al., 2017; Yao et al., 2019) or hardware (Cao et al., 2019) in order to accelerate matrix multi- plication, therefore limiting their application to a broad set …

Content truncated to 500 words