FastBERT: a Self-distilling BERT with Adaptive Inference Time

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6035–6044 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6035 FastBERT: a Self-distilling BERT with Adaptive Inference Time Weijie Liu1,2, Peng Zhou2, Zhiruo Wang3, Zhe Zhao2, Haotang Deng2 and Qi Ju2,∗ 1Peking University, Beijing, China 2Tencent Research, Beijing, China 3Beijing Normal University, Beijing, China dataliu@pku.edu.cn, {rickzhou, nlpzhezhao, haotangdeng, damonju}@tencent.com, SherronWang@gmail.com Abstract Pre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can hardly be readily implemented with limited re- sources. To improve their efficiency with an as- sured model performance, we propose a novel speed-tunable FastBERT with adaptive infer- ence time. The speed at inference can be flex- ibly adjusted under varying demands, while redundant calculation of samples is avoided. Moreover, this model adopts a unique self- distillation mechanism at fine-tuning, further enabling a greater computational efficacy with minimal loss in performance. Our model achieves promising results in twelve English and Chinese datasets. It is able to speed up by a wide range from 1 to 12 times than BERT if given different speedup thresholds to make a speed-performance tradeoff. 1 Introduction Last two years have witnessed significant improve- ments brought by language pre-training, such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018), and XLNet (Yang et al., 2019). By pre- training on unlabeled corpus and fine-tuning on la- beled ones, BERT-like models achieved huge gains on many Natural Language Processing tasks. Despite this gain in accuracy, these models have greater costs in computation and slower speed at in- ference, which severely impairs their practicalities. Actual settings, especially with limited time and resources in the industry, can hardly enable such models into operation. For example, in tasks like sentence matching and text classification, one often requires to process billions of requests per second. What’s more, the number of requests varies with time. In the case of an online shopping site, the ∗Corresponding author: Qi Ju (damonju@tencent.com) number of requests during the holidays is five to ten times more than that of the workdays. A large number of servers need to be deployed to enable BERT in industrial settings, and many spare servers need to be reserved to cope with the peak period of requests, demanding huge costs. To improve their usability, many attempts in model acceleration have been made, such as quan- tinization (Gong et al., 2014), weights pruning (Han et al., 2015), and knowledge distillation (KD) (Romero et al., 2014). As one of the most popular methods, KD requires additional smaller student models that depend entirely on the bigger teacher model and trade task accuracy for ease in computa- tion (Hinton et al., 2015). Reducing model sizes to achieve acceptable speed-accuracy balances, how- ever, can only solve the problem halfway, for the model is still set as fixated, rendering them unable to cope with drastic changes in request amount. By inspecting many NLP datasets (Wang et …

Content truncated to 500 words