DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2246 DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference Ji Xin1,2, Raphael Tang1,2, Jaejun Lee1, Yaoliang Yu1,2, and Jimmy Lin1,2 1David R. Cheriton School of Computer Science, University of Waterloo 2Vector Institute for Artificial Intelligence {ji.xin,r33tang,j474lee,yaoliang.yu,jimmylin}@uwaterloo.ca Abstract Large-scale pre-trained language models such as BERT have brought significant improve- ments to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real- time applications. We propose a simple but ef- fective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ∼40% inference time with minimal degradation in model quality. Fur- ther analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is avail- able at https://github.com/castorini/ DeeBERT. 1 Introduction Large-scale pre-trained language models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2019), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019) have brought significant improvements to natural lan- guage processing (NLP) applications. Despite their power, they are notorious for being enormous in size and slow in both training and inference. Their long inference latencies present challenges to de- ployment in real-time applications and hardware- constrained edge devices such as mobile phones and smart watches. To accelerate inference for BERT, we propose DeeBERT: Dynamic early exiting for BERT. The inspiration comes from a well-known observa- tion in the computer vision community: in deep convolutional neural networks, higher layers typi- cally produce more detailed and finer-grained fea- tures (Zeiler and Fergus, 2014). Therefore, we Figure 1: DeeBERT model overview. Grey blocks are transformer layers, orange circles are classification lay- ers (off-ramps), and blue arrows represent inference samples exiting at different layers. hypothesize that, for BERT, features provided by the intermediate transformer layers may suffice to classify some input samples. DeeBERT accelerates BERT inference by insert- ing extra classification layers (which we refer to as off-ramps) between each transformer layer of BERT (Figure 1). All transformer layers and off- ramps are jointly fine-tuned on a given downstream dataset. At inference time, after a sample goes through a transformer layer, it is passed to the fol- lowing off-ramp. If the off-ramp is confident of the prediction, the result is returned; otherwise, the sample is sent to the next transformer layer. In this paper, we conduct experiments on BERT and RoBERTa with six GLUE datasets, showing that DeeBERT is capable of accelerating model in- ference by up to ∼40% with minimal model quality degradation on downstream tasks. Further analy- ses reveal interesting patterns in the models’ trans- former layers, as well as redundancy in both BERT and RoBERTa. 2 Related Work BERT and RoBERTa …

Content truncated to 500 words