Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 719–730 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 719 Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior Zi Lin∗ Google Research lzi@google.com Jeremiah Zhe Liu† Google Research & Harvard University jereliu@google.com Zi Yang Google Research ziy@google.com Nan Hua Google Research nhua@google.com Dan Roth University of Pennsylvania danroth@seas.upenn.edu Abstract Traditional (unstructured) pruning methods for a Transformer model focus on regularizing the individual weights by penalizing them to- ward zero. In this work, we explore spectral- normalized identity priors (SNIP), a structured pruning approach that penalizes an entire resid- ual module in a Transformer model toward an identity mapping. Our method identifies and discards unimportant non-linear mappings in the residual connections by applying a thresh- olding operator on the function norm. It is applicable to any structured module, includ- ing a single attention head, an entire attention block, or a feed-forward subnetwork. Further- more, we introduce spectral normalization to stabilize the distribution of the post-activation values of the Transformer layers, further im- proving the pruning effectiveness of the pro- posed methodology. We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective prun- ing results while maintaining comparable per- formance. Specifically, we improve the perfor- mance over the state-of-the-art by 0.5 to 1.0% on average at 50% compression ratio. 1 Introduction Natural Language Processing (NLP) has recently achieved great success by using the Transformer- based pre-trained models (Radford et al., 2019; Devlin et al., 2018; Yang et al., 2019; Clark et al., 2020). However, these models often consume con- siderable storage, memory bandwidth, and compu- tational resource. To reduce the model size and in- crease the inference throughput, compression tech- niques such as knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Tang et al., 2019; Jiao et al., 2019; Sun et al., 2020) and weight pruning (Guo ∗Work done as part of the Google AI Residency. †Work done at Google Research. 20 40 60 80 100 Remaining weights % 70 75 80 85 Accuracy MNLI-mm SNIP (Ours) Iterative Pruning DistilBERT BERT-PKD Figure 1: Comparison of selected knowledge distil- lation methods (DistilBERT (Sanh et al., 2019) and BERT-PKD (Sun et al., 2019)) and iterative pruning methods (Iterative Pruning (Guo et al., 2019) and our proposed method) in terms of accuracy at various com- pression rate using MNLI test set. knowledge distil- lation methods require re-distillation from the teacher to get each single data point, whereas iterative pruning methods can produce continuous curves at once. et al., 2019; Wang et al., 2019; Gordon et al., 2020; Sanh et al., 2020) have recently been developed. Knowledge distillation methods require the spec- ification of a student network with a smaller archi- tecture, which often has to be identified by a tedious sequence process of trial-and-error based decisions. By comparison, the iterative pruning methods grad- ually prune the redundant model weights or layers from the full-size model, and provide a full picture of the trade-off …

Content truncated to 500 words