A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2013–2023 June 6–11, 2021. ©2021 Association for Computational Linguistics 2013 A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models Kaiyuan Liao†∗, Yi Zhang‡∗, Xuancheng Ren‡, Qi Su‡§, Xu Sun†‡, Bin He⋆ † Center for Data Science, Peking University ‡ MOE Key Laboratory of Computational Linguistics, School of EECS, Peking University § School of Foreign Languages, Peking University ⋆ Huawei Noah’s Ark Lab {kyliao,zhangyi16,renxc,sukia,xusun}@pku.edu.cn hebin.nlp@huawei.com Abstract Early exit mechanism aims to accelerate the inference speed of large-scale pre-trained lan- guage models. The essential idea is to exit early without passing through all the inference layers at the inference stage. To make accu- rate predictions for downstream tasks, the hi- erarchical linguistic information embedded in all layers should be jointly considered. How- ever, much of the research up to now has been limited to use local representations of the exit layer. Such treatment inevitably loses infor- mation of the unused past layers as well as the high-level features embedded in future lay- ers, leading to sub-optimal performance. To address this issue, we propose a novel Past- Future method to make comprehensive pre- dictions from a global perspective. We first take into consideration all the linguistic infor- mation embedded in the past layers and fur- ther engage the future information which is originally inaccessible for predictions. Exten- sive experiments demonstrate that our method outperforms previous early exit methods by a large margin, yielding better and robust perfor- mance1. 1 Introduction Pre-trained language models (PLMs), e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019), have obtained remarkable success in a wide range of NLP tasks. Despite their impressive performance, PLMs are usually associated with large memory requirement and high computational cost. Such drawbacks slow down the inference and further encumber the application of PLMs in the scenarios where inference time and computation budget are restricted. To address this issue, a growing number of stud- ies focusing on improving model efficiency have ∗Equal contribution 1The code is available at https://github.com/ lancopku/Early-Exit emerged recently. Particularly, Kaya et al. (2019) point out that the current over-parameterized mod- els conduct excessive computation for simple in- stances, which is actually undesirable and compu- tationally wasteful. In light of this observation, an increasing amount of work seeks various early exit methods, of which the basic idea is to exit early without passing through the entire model during inference. Concretely, for NLP tasks, they couple branch classifiers with each layer of the pre-trained language models and stop forward propagation at an intermediate layer. Then the current branch classifier makes a prediction based on the represen- tation of the token that is used as the aggregated sequence representation for classification tasks and is referred to as the state of the layer in this work. However, existing work on early exit has two major drawbacks. First, existing work (Xin et al., …

Content truncated to 500 words