Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks

Proceedings of the 29th International Conference on Computational Linguistics, pages 2861–2872 October 12–17, 2022. 2861 Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks Rajiv Movva*123, Jinhao Lei*2, Shayne Longpre23, Ajay Gupta2, and Chris DuBois2 *Equal Contribution 1Cornell Tech, rm868@cornell.edu 2Apple, {jlei2, ajay_gupta2, cdubois}@apple.com 3Massachusetts Institute of Technology, slongpre@mit.edu Abstract Quantization, knowledge distillation, and mag- nitude pruning are among the most popular methods for neural network compression in NLP. Independently, these methods reduce model size and can accelerate inference, but their relative benefit and combinatorial inter- actions have not been rigorously studied. For each of the eight possible subsets of these tech- niques, we compare accuracy vs. model size tradeoffs across six BERT architecture sizes and eight GLUE tasks. We find that quan- tization and distillation consistently provide greater benefit than pruning. Surprisingly, ex- cept for the pair of pruning and quantization, using multiple methods together rarely yields diminishing returns. Instead, we observe com- plementary and super-multiplicative reductions to model size. Our work quantitatively demon- strates that combining compression methods can synergistically reduce model size, and that practitioners should prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to maximize accuracy vs. model size tradeoffs. 1 Introduction As increasingly large models dominate Natural Language Processing (NLP) benchmarks, model compression techniques have grown in popularity (Gupta and Agrawal, 2020; Rogers et al., 2020; Ganesh et al., 2021). For example, quantization (Shen et al., 2020; Zafrir et al., 2019; Jacob et al., 2018) lowers bit precision of network weights to reduce memory usage and accelerate inference (Krashinsky et al., 2020). Knowledge distillation (KD; Hinton et al. (2015)), which trains a student neural network using the logits (or representations) of a teacher network, is used widely to transfer knowledge to smaller models (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2019, 2020). Pruning identifies weights which can be omitted at test time without significantly degrading performance. Some pruning methods remove individual weights accord- ing to magnitudes or other heuristics (Gordon et al., 2020; Chen et al., 2020; Sanh et al., 2020), while others remove structured blocks of weights or en- tire attention heads (Wang et al., 2020; Hou et al., 2020; Voita et al., 2019; Michel et al., 2019). Recent work has begun combining these com- pression methods for improved results. Sanh et al. (2020), Zhang et al. (2020), and Bai et al. (2021) have used knowledge distillation with pruning or low-bit quantization to fine-tune BERT. As practi- tioners look to combine methods more generally, new research is needed to compare their empiri- cal value and study interactions. This work ad- dresses the questions: (1) Which popular compres- sion methods or combinations of methods are usu- ally most effective? (2) When combining methods, are their benefits complementary or diminishing? We address these questions by computing ac- curacy vs. model size tradeoff curves for six pre- trained BERT sizes fine-tuned on eight GLUE tasks (Wang et al., 2019b), applying each of eight possi- ble subsets of quantization-aware-training (QAT), knowledge distillation …

Content truncated to 500 words