Accelerating Inference for Pretrained Language Models by Unified Multi-Perspective Early Exiting

Proceedings of the 29th International Conference on Computational Linguistics, pages 4677–4686 October 12–17, 2022. 4677 Accelerating Inference for Pretrained Language Models by Unified Multi-Perspective Early Exiting Jun Kong †, Jin Wang†∗, Liang-Chih Yu‡∗ and Xuejie Zhang† †School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China ‡Department of Information Management, Yuan Ze University, Taiwan Contact:wangjin@ynu.edu.cn, lcyu@saturn.yzu.edu.tw Abstract Conditional computation algorithms, such as the early exiting (EE) algorithm, can be ap- plied to accelerate the inference of pretrained language models while maintaining competi- tive performance on resource-constrained de- vices. However, this approach is only applied to the vertical architecture to decide which layers should be used for inference. Con- versely, the operation of the horizontal per- spective is ignored, and the determination of which tokens in each layer should participate in the computation fails, leading to a high re- dundancy for adaptive inference. To address this limitation, a unified horizontal and ver- tical multi-perspective early exiting (MPEE) framework is proposed in this study to acceler- ate the inference of transformer-based models. Specifically, the vertical architecture uses recy- cling EE classifier memory and weighted self- distillation to enhance the performance of the EE classifiers. Then, the horizontal perspec- tive uses recycling class attention memory to emphasize the informative tokens. Conversely, the tokens with less information are truncated by weighted fusion and isolated from the fol- lowing computation. Based on this, both hor- izontal and vertical EE are unified to obtain a better tradeoff between performance and effi- ciency. Extensive experimental results show that MPEE can achieve higher acceleration in- ference with competent performance than ex- isting competitive methods. The code for this paper is available at: https://github. com/JunKong5/MPEE. 1 Introduction Pretrained language models (PLMs) (Devlin et al., 2019; Lan et al., 2019; Liu et al., 2019) have shown promising performance in many natural language processing tasks. The success of PLMs incurs com- putational consumption and long inference latency, which prevents these models from being deployed ∗Corresponding authors. Figure 1: Tokens (cosine similarity) of different layers. on resource-constrained devices, such as edge de- vices, or in time-sensitive scenarios. To address these issues, model compression tech- niques (Lin et al., 2020), including knowledge dis- tillation (Hinton et al., 2015; Jiao et al., 2020), prun- ing (Sanh et al., 2020; Michel et al., 2019) and quantization (Zafrir et al., 2019), have been applied to PLMs to accelerate inference. However, these methods permanently remove some components of the model, leading to an inevitable decline in performance. Additionally, the complexity of these models cannot be adjusted based on the require- ments of different hardware since the compression is implemented before the deployment. Conversely, several studies have suggested the use of condi- tional computation algorithms, such as the early exiting (EE) algorithm (Schwartz et al., 2020; Xin et al., 2020), in which each input sample uses a dif- ferent part of the model so that the computation or latency is reduced on average. Based on this, an EE classifier is added between transformer layers. The shallow layer …

Content truncated to 500 words