Block Pruning For Faster Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10619–10629 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10619 Block Pruning For Faster Transformers François Lagunas, Ella Charlaix, Victor Sanh, Alexander M. Rush Hugging Face {francois, ella, victor, sasha}@huggingface.co Abstract Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models. Pruning methods have proven to be an effective way of reducing model size, whereas distillation methods are proven for speeding up inference. We introduce a block pruning approach targeting both small and fast mod- els. Our approach extends structured meth- ods by considering blocks of any size and in- tegrates this structure into the movement prun- ing paradigm for fine-tuning. We find that this approach learns to prune out full com- ponents of the underlying model, such as at- tention heads. Experiments consider classifi- cation and generation tasks, yielding among other results a pruned model that is a 2.4x faster, 74% smaller BERT on SQuAD v1, with a 1% drop on F1, competitive both with dis- tilled models in speed and pruned models in size. 1 Introduction Pre-trained transformer models are the standard for NLP tasks in both classification and genera- tion tasks (Devlin et al., 2019; Lewis et al., 2020). The recent trend is for models to continue to grow in size while yielding improved performance on standard benchmarks (Rosset, 2020). This develop- ment highlights the need to reduce the storage size and increase the efficiency of pre-trained models. Pruning methods have shown to be extremely effective at reducing the storage size of models fine-tuned for a specific task. Approaches such as magnitude pruning (Han et al., 2015), L0 regular- ization (Louizos et al., 2018), lottery ticket hypoth- esis (Frankle and Carbin, 2018), diff pruning (Guo et al., 2020), and movement pruning (Sanh et al., 2020) have demonstrated remarkable reductions in model size. Movement pruning produces 77% sav- ings in parameter storage for a 1% drop in accuracy on SQuAD v1.1. However, these models yield very little actual efficiency benefits, as to run them in standard hardware often requires reconstructing the original dense shape. On the other hand distillation methods have been more effective at producing faster models as has been shown by DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019) or MobileBERT (Sun et al., 2020). These approaches utilize targeted dis- tillation to produce smaller models with a dense structure that is fast on standard hardware. How- ever without careful engineering and size selection these models are much larger than pruned ones. In this work, we target closing this gap through block pruning. Unlike pruning individual param- eters, this approach encourages pruning that can be optimized on dense hardware. It is a less rigid approach than row or column-based pruning typ- ically used in structured approaches (McCarley, 2019), which have been difficult to apply effec- tively to transformers. We integrate this approach with Movement pruning (Sanh et al., 2020), a sim- …

Content truncated to 500 words