BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7859–7869, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7859 BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu1∗, Wangchunshu Zhou2∗, Tao Ge3, Furu Wei3, Ming Zhou3 1 University of California, San Diego 2 Beihang University 3 Microsoft Research Asia 1 cxu@ucsd.edu 2 zhouwangchunshu@buaa.edu.cn 3 {tage,fuwei,mingzhou}@microsoft.com Abstract In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact sub- stitutes. Then, we randomly replace the origi- nal modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively in- crease the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the orig- inal and compact models. Compared to the previous knowledge distillation approaches for BERT compression, our approach does not in- troduce any additional loss function. Our ap- proach outperforms existing knowledge distil- lation approaches on GLUE benchmark, show- ing a new perspective of model compression.1 1 Introduction With the prevalence of deep learning, many huge neural models have been proposed and achieve state-of-the-art performance in various fields (He et al., 2016; Vaswani et al., 2017). Specifically, in Natural Language Processing (NLP), pretrain- ing and fine-tuning have become the new norm of most tasks. Transformer-based pretrained mod- els (Devlin et al., 2019; Liu et al., 2019b; Yang et al., 2019; Song et al., 2019; Dong et al., 2019) have dominated the field of both Natural Language Understanding (NLU) and Natural Language Gen- eration (NLG). These models benefit from their “overparameterized” nature (Nakkiran et al., 2020) and contain millions or even billions of parameters, making it computationally expensive and ineffi- cient considering both memory consumption and ∗ Equal contribution. Work done during these two authors’ internship at Microsoft Research Asia. 1The code and pretrained model are available at https: //github.com/JetRunner/BERT-of-Theseus high latency. This drawback enormously hinders the applications of these models in production. To resolve this problem, many techniques have been proposed to compress a neural network. Gen- erally, these techniques can be categorized into Quantization (Gong et al., 2014), Weights Prun- ing (Han et al., 2016) and Knowledge Distillation (KD) (Hinton et al., 2015). Among them, KD has received much attention for compressing pretrained language models. KD exploits a large teacher model to “teach” a compact student model to mimic the teacher’s behavior. In this way, the knowledge embedded in the teacher model can be transferred into the smaller model. However, the retained per- formance of the student model relies on a well- designed distillation loss function which forces the student model to behave as the teacher. Recent studies on KD (Sun et al., 2019; Jiao et al., 2019) even leverage more sophisticated model-specific distillation loss functions for better performance. Different from previous KD studies which ex- plicitly exploit a distillation loss to minimize the distance between the teacher …

Content truncated to 500 words