Accelerating BERT Inference for Sequence Labeling via Early-Exit

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 189–199 August 1–6, 2021. ©2021 Association for Computational Linguistics 189 Accelerating BERT Inference for Sequence Labeling via Early-Exit Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu∗, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University {lixn20, yfshao19, txsun19, hyan19, xpqiu, xjhuang}@fudan.edu.cn Abstract Both performance and efficiency are crucial factors for sequence labeling tasks in many real-world scenarios. Although the pre-trained models (PTMs) have significantly improved the performance of various sequence labeling tasks, their computational cost is expensive. To alleviate this problem, we extend the re- cent successful early-exit mechanism to accel- erate the inference of PTMs for sequence label- ing tasks. However, existing early-exit mech- anisms are specifically designed for sequence- level tasks, rather than sequence labeling. In this paper, we first propose SENTEE: a sim- ple extension of SENTence-level Early-Exit for sequence labeling tasks. To further reduce computational cost, we also propose TOKEE: a TOKen-level Early-Exit mechanism that al- lows partial tokens to exit early at different layers. Considering the local dependency in- herent in sequence labeling, we employed a window-based criterion to decide for a token whether or not to exit. The token-level early- exit brings the gap between training and infer- ence, so we introduce an extra self-sampling fine-tuning stage to alleviate it. The extensive experiments on three popular sequence label- ing tasks show that our approach can save up to 66%∼75% inference cost with minimal perfor- mance degradation. Compared with compet- itive compressed models such as DistilBERT, our approach can achieve better performance under the same speed-up ratios of 2×, 3×, and 4×.1 1 Introduction Sequence labeling plays an important role in natu- ral language processing (NLP). Many NLP tasks can be converted to sequence labeling tasks, such as named entity recognition, part-of-speech tagging, ∗Corresponding author. 1Our implementation is publicly available at https:// github.com/LeeSureman/Sequence-Labeling-Early-Exit. Chinese word segmentation and Semantic Role La- beling. These tasks are usually fundamental and highly time-demanding, therefore, apart from per- formance, their inference efficiency is also very important. The past few years have witnessed the prevailing of pre-trained models (PTMs) (Qiu et al., 2020) on various sequence labeling tasks (Nguyen et al., 2020; Ke et al., 2020; Tian et al., 2020; Mengge et al., 2020). Despite their significant improve- ments on sequence labeling, they are notorious for enormous computational cost and slow inference speed, which hinders their utility in real-time sce- narios or mobile-device scenarios. Recently, early-exit mechanism (Liu et al., 2020; Xin et al., 2020; Schwartz et al., 2020; Zhou et al., 2020) has been introduced to accelerate inference for large-scale PTMs. In their methods, each layer of the PTM is coupled with a classifier to pre- dict the label for a given instance. At inference stage, if the prediction is confident 2 enough at an earlier time, it is allowed to exit without pass- ing through the …

Content truncated to 500 words