EBERT: Efficient BERT Inference with Dynamic Structured Pruning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4814–4823 August 1–6, 2021. ©2021 Association for Computational Linguistics 4814 EBERT: Efficient BERT Inference with Dynamic Structured Pruning Zejian Liu1,2, Fanrong Li1,2, Gang Li1 and Jian Cheng1,2 1National Laboratory of Pattern Recognition, Institute of Automation, CAS 2School of Future Technology, University of Chinese Academy of Sciences {liuzejian2018, lifanrong2017}@ia.ac.cn, gangli0426@gmail.com, jcheng@nlpr.ia.ac.cn Abstract Pruning has been demonstrated as an effective way of reducing computational complexity for deep networks, especially CNNs for computer vision tasks. In this paper, we investigate the opportunity to accelerate the inference of large-scale pre-trained language model via pruning. We propose EBERT, a dynamic struc- tured pruning algorithm for efficient BERT in- ference. Unlike previous methods that ran- domly prune the model weights for static in- ference, EBERT dynamically determines and prunes the unimportant heads in multi-head self-attention layers and the unimportant struc- tured computations in feed-forward network for each input sample at run-time. Experimen- tal results show that our proposed EBERT out- performs other state-of-the-art methods on dif- ferent tasks. 1 Introduction In the last few years, transformer-based (Vaswani et al., 2017) large-scale pre-trained language mod- els, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020), have achieved state-of-the-art results on many NLP tasks, including language understand- ing, question answering, and reading comprehen- sion. Most recently, researchers also successfully applied transformer-based models to computer vi- sion tasks, achieving comparable or superior perfor- mance compared to traditional convolutional net- works. For example, Carion et al. (2020) propose detection transformer (DETR) for object detection, Dosovitskiy et al. (2021) design a transformer- based model, namely Vision Transformer (ViT), for image classification. However, due to the no- table computational complexity and memory foot- print, it is difficult for these models to deploy on hardware platforms under moderate computing and resource budget. Therefore, how to reduce model complexity to enable efficient inference for large- scale pre-trained language models is a critical issue. Pruning is a commonly used technique for net- work compression, which has been widely explored to reduce computation and storage requirements of convolutional neural networks for computer vision tasks (Han et al., 2015, 2016; Li et al., 2016). How- ever, can transformer-based models benefit from pruning? Michel et al. (2019) observe that a large percentage of attention heads can be removed with negligible performance drop, which indicates that the importance of different heads in same layer is different. Sanh et al. (2020) propose a simple, deter- ministic first-order weight pruning method which can prune lots of parameters with minimal accuracy loss. Although these methods are able to reduce the memory footprint, they cannot achieve real per- formance gain on general-purpose hardware, such as GPGPU, due to the unstructured sparsity after pruning. Adaptive inference strategy is also proposed to accelerate the inference of BERT. It is based on two observations: 1) the input samples usually have dif- ferent levels of difficulty. For a given model, it may over-calculate the simple samples while fail …

Content truncated to 500 words