Aligned Weight Regularizers for Pruning Pretrained Neural Networks

Findings of the Association for Computational Linguistics: ACL 2022, pages 3391 - 3401 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Aligned Weight Regularizers for Pruning Pretrained Neural Networks James O’ Neill and Sourav Dutta and Haytham Assem Huawei Research Center, Dublin, Ireland {james.o.neil,sourav.dutta2,haytham.assem}@huawei.com Abstract While various avenues of research have been explored for iterative pruning, little is known what effect pruning has on zero-shot test per- formance and its potential implications on the choice of pruning criteria. This pruning setup is particularly important for cross-lingual mod- els that implicitly learn alignment between language representations during pretraining, which if distorted via pruning, not only leads to poorer performance on language data used for retraining but also on zero-shot languages that are evaluated. In this work, we show that there is a clear performance discrepancy in magnitude-based pruning when comparing standard supervised learning to the zero-shot setting. From this finding, we propose two weight regularizers that aim to maximize the alignment between units of pruned and un- pruned networks to mitigate alignment distor- tion in pruned cross-lingual models and per- form well for both non zero-shot and zero-shot settings. We provide experimental results on cross-lingual tasks for the zero-shot setting us- ing XLM-RoBERTaBase, where we also find that pruning has varying degrees of representa- tional degradation depending on the language corresponding to the zero-shot test set. This is also the first study that focuses on cross-lingual language model compression. 1 Introduction Deep neural networks (DNNs) have grown increas- ingly large in the recent years. This has led to mod- els requiring more storage requirements, more re- sources for training and inference (e.g., GPUs and TPUs), longer compute times and larger carbon footprints. This is largely due to the rise of masked self-supervised learning (SSL) which trains DNNs (e.g., Transformers in NLP) on a large collection of samples that do not have task labels but instead use a subset of the inputs as labels. Given the afore- mentioned challenges, it has become more difficult for machine learning practitioners to use these SSL pretrained models for fine-tuning on downstream tasks. While training tricks such as effective batch sizes, gradient accumulation and dynamic learning rate schedules (Howard and Ruder, 2018) have im- proved the efficiency of fine-tuning DNNs under resource constraints, it can still come at a cost, e.g. gradient accumulation leads to less updates. Pruning (LeCun et al., 1990; Reed, 1993) is a type of model compression method (Buciluˇa et al., 2006) that aims to address these shortcomings by zeroing out a subset of weights in the DNN, while maintaining performance close to the original model. Retraining is often carried out directly after each pruning step to recover from pruning induced performance drops. This process is referred to as iterative pruning. Although, iterative pruning has been extensively studied in the SSL setting (Has- sibi and Stork, 1993; Han et al., 2016; Ding et al., 2018) and the transfer learning setting (Molchanov et al., 2016; Gordon et al., 2020; Sanh et al., 2020), little …

Content truncated to 500 words