LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

Proceedings of the 28th International Conference on Computational Linguistics, pages 3225–3234 Barcelona, Spain (Online), December 8-13, 2020 3225 LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression Yihuan Mao1,∗, Yujing Wang2,3,†, Chufan Wu1, Chen Zhang2, Yang Wang2 Quanlu Zhang2, Yaming Yang2, Yunhai Tong3, Jing Bai2 1Tsinghua University 2Microsoft Research Asia 3Key Laboratory of Machine Perception, MOE, School of EECS, Peking University maoyh20@mails.tsinghua.edu.cn, {yujwang,yhtong}@pku.edu.cn, chufanwu15@gmail.com {yujwang,zhac,t-yangwa,yayaming,quzha,jbai}@microsoft.com Abstract BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests. Existing solutions leverage knowledge distilla- tion frameworks to learn smaller models that imitate the behaviors of BERT. However, the train- ing procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a hybrid solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude. 1 Introduction The pre-trained language model, BERT (Devlin et al., 2018) has led to a big breakthrough in various kinds of natural language understanding tasks. Ideally, people can start from a pre-trained BERT check- point and fine-tune it on a specific downstream task. However, the original BERT models are memory- exhaustive and latency-prohibitive to be served in embedded devices or CPU-based online environments. As the memory and latency constraints vary in different scenarios, the pre-trained BERT model should be adaptive to different requirements with accuracy retained to the largest extent. Existing BERT-oriented model compression solutions largely depend on knowledge distillation (Hinton et al., 2015), which is inefficient and resource-consuming because a large training corpus is required to learn the behaviors of a teacher. For example, DistilBERT (Sanh et al., 2019) is re-trained on the same corpus as pre-training a vanilla BERT from scratch; and TinyBERT (Jiao et al., 2019) utilizes expensive data augmentation to fit the distillation target. The costs of these model compression methods are as large as pre-training, which are unaffordable for low-resource settings. Therefore, it is straight-forward to ask, can we design a lightweight method to generate adaptive models with comparable accuracy using significantly less time and resource consumption? In this paper, we propose LadaBERT (Lightweight adaptation of BERT through hybrid model com- pression) to tackle this problem. Specifically, LadaBERT is based on an iterative hybrid model com- pression framework consisting of weighting pruning, matrix factorization and knowledge distillation. Initially, the architecture and weights of student model are inherited from the BERT teacher. In each iteration, the student model is first compressed by a small ratio based on weight pruning and matrix factorization, and is then fine-tuned under the guidance of teacher model through knowledge distillation. Because …

Content truncated to 500 words