Patient Knowledge Distillation for BERT Model Compression

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4323–4332, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4323 Patient Knowledge Distillation for BERT Model Compression Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu Microsoft Dynamics 365 AI Research {Siqi.Sun,Yu.Cheng,Zhe.Gan,jingjl}@microsoft.com Abstract Pre-trained language models such as BERT have proven to be highly effective for natu- ral language processing (NLP) tasks. How- ever, the high demand for computing resources in training such models hinders their applica- tion in practice. In order to alleviate this re- source hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from pre- vious knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple interme- diate layers of the teacher model for incremen- tal knowledge extraction, following two strate- gies: (i) PKD-Last: learning from the last k layers; and (ii) PKD-Skip: learning from ev- ery k layers. These two patient distillation schemes enable the exploitation of rich infor- mation in the teacher’s hidden layers, and en- courage the student model to patiently learn from and imitate the teacher through a multi- layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training ef- ficiency, without sacrificing model accuracy.1 1 Introduction Language model pre-training has proven to be highly effective in learning universal language representations from large-scale unlabeled data. ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2018) have achieved great success in many NLP tasks, such as sentiment classification (Socher et al., 2013), natu- ral language inference (Williams et al., 2017), and question answering (Lai et al., 2017). 1Code will be avialable at https://github.com/ intersun/PKD-for-BERT-Model-Compression. Despite its empirical success, BERT’s compu- tational efficiency is a widely recognized issue be- cause of its large number of parameters. For exam- ple, the original BERT-Base model has 12 layers and 110 million parameters. Training from scratch typically takes four days on 4 to 16 Cloud TPUs. Even fine-tuning the pre-trained model with task- specific dataset may take several hours to finish one epoch. Thus, reducing computational costs for such models is crucial for their application in prac- tice, where computational resources are limited. Motivated by this, we investigate the redun- dancy issue of learned parameters in large-scale pre-trained models, and propose a new model compression approach, Patient Knowledge Distil- lation (Patient-KD), to compress original teacher (e.g., BERT) into a lightweight student model without performance sacrifice. In our approach, the teacher model outputs probability logits and predicts labels for the training samples (extend- able to additional unannotated samples), and the student model learns from the teacher network to mimic the teacher’s prediction. Different from previous knowledge distillation methods (Hinton et al., 2015; Sau and Balasub- ramanian, 2016; …

Content truncated to 500 words