Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10653–10659 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10653 Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression Canwen Xu1∗, Wangchunshu Zhou2∗, Tao Ge3, Ke Xu4, Julian McAuley1, Furu Wei3 1 University of California, San Diego 2 Stanford University 3 Microsoft Research Asia 4 Beihang University 1{cxu,jmcauley}@ucsd.edu, 2wcszhou@stanford.edu 3{tage,fuwei}@microsoft.com, 4kexu@nlsde.buaa.edu.cn Abstract Recent studies on compression of pretrained language models (e.g., BERT) usually use pre- served accuracy as the metric for evaluation. In this paper, we propose two new metrics, la- bel loyalty and probability loyalty that mea- sure how closely a compressed model (i.e., stu- dent) mimics the original model (i.e., teacher). We also explore the effect of compression with regard to robustness under adversarial attacks. We benchmark quantization, pruning, knowl- edge distillation and progressive module re- placing with loyalty and robustness. By com- bining multiple compression techniques, we provide a practical strategy to achieve better accuracy, loyalty and robustness.1 1 Introduction Recently, many large pretrained language models (PLMs, Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Shoeybi et al., 2019; Raffel et al., 2020) have been proposed for a variety of Natu- ral Language Processing (NLP) tasks. However, as pointed out in recent studies (Strubell et al., 2019; Schwartz et al., 2020a; Bender et al., 2021), these models suffer from computational inefficiency and high ecological cost. Many attempts have been made to address this problem, including quantiza- tion (Zafrir et al., 2019; Shen et al., 2020), prun- ing (Michel et al., 2019; Sanh et al., 2020), knowl- edge distillation (KD) (Sanh et al., 2019; Sun et al., 2019, 2020; Turc et al., 2019; Jiao et al., 2020; Wang et al., 2020; Zhou et al., 2021) and progres- sive module replacing (Xu et al., 2020). BERT (Devlin et al., 2019) is a representative PLM. Many works compressing BERT use pre- served accuracy with computational complexity (e.g., speed-up ratio, FLOPS, number of param- eters) as metrics to evaluate compression. This Asia.∗Equal Contribution. Work done at Microsoft Research 1Our code is available at https://github.com/ JetRunner/beyond-preserved-accuracy. Label Predicted probability distribution Original Model Compressed Model Compression Quantization Knowledge Distillation Pruning Module Replacing Probability Loyalty (0.6, 0.25, 0.15) Label Loyalty Neutral Entailment (0.7, 0.2, 0.1) Robustness Adversarial Attack Figure 1: Three metrics to evaluate the compressed models beyond preserved accuracy. For each input, la- bel and probability loyalty measure the shift of label and predicted probability distribution, respectively. Ro- bustness measures the performance of the compressed model under adversarial attacks. evaluation scheme is far from perfect: (1) Pre- served accuracy cannot reflect how alike the teacher and student2 models behave. This can be prob- lematic when applying compression techniques in production (to be detailed in Section 3). (2) Us- ing preserved accuracy to evaluate models com- pressed with more data or data augmentation (Jiao et al., 2020) can be misleading, since one cannot tell whether the improvement should be attributed to the innovation of the compression …

Content truncated to 500 words