Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3187–3199 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3187 Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning Bingbing Li ∗1, Zhenglun Kong ∗2, Tianyun Zhang 3, Ji Li 4, Zhengang Li 2, Hang Liu 5, Caiwen Ding 1 1University of Connecticut, 2Northeastern University, 3Syracuse University, 4Microsoft Corporation, 5Stevens Institute of Technology {bingbing.li, caiwen.ding}@uconn.edu, {kong.zhe, li.zhen}@northeastern.edu, tzhan120@syr.edu, changzhouliji@gmail.com, hliu77@stevens.edu Abstract Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and com- putational speed on hardware platforms have impeded the popularity of pre-trained mod- els, especially in the era of edge comput- ing. In this work, we propose an efficient transformer-based large-scale language repre- sentation using hardware-friendly block struc- ture pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly re- duced weight storage and computation, the proposed approach achieves high compression rates. Experimental results on different mod- els (BERT, RoBERTa, and DistilBERT) on the General Language Understanding Evalua- tion (GLUE) benchmark tasks show that we achieve up to 5.0× with zero or minor ac- curacy degradation on certain task(s). Our proposed method is also orthogonal to exist- ing compact pre-trained language models such as DistilBERT using knowledge distillation, since a further 1.79× average compression rate can be achieved on top of DistilBERT with zero or minor accuracy degradation. It is suit- able to deploy the final compressed model on resource-constrained edge devices. We share the related codes and models at: https://bi t.ly/3cvs2N2 1 Introduction Transformer-based language model pre-training has proven to be highly effective in learning univer- sal language representations from large-scale unla- beled data and being fine-tuned to adapt to down- stream tasks (Peters et al., 2018; Sun et al., 2019). Representative works such as BERT (Devlin et al., 2018), XLNet (Yang et al., 2019), RoBERTa (Liu ∗These authors contributed equally et al., 2019b), MT-DNN (Liu et al., 2019a), AL- BERT (Lan et al., 2019), GPT-2 (Radford et al.), and UniLMv2 (Bao et al., 2020) have substantially advanced the state-of-the-art across a variety of downstream tasks, such as text classification, natu- ral language inference, and question answering. Despite its success in performance improve- ment in natural language understanding and gen- eration, the computational cost and data storage of Transformer-based pre-trained language model are two widely recognized concerns due to Trans- former’s deep architecture and rich parameters. These models typically contain several hundred million parameters. The recent released research models even reach multi-billion parameters, such as MegatronLM (8.3 billion parameters) (Shoeybi et al., 2019), Turing-NLG (17 billion parame- ters) (Microsoft, 2020) and GPT-3 (175 billion pa- rameters) (Brown et al., 2020), which require more advanced computing facility. Hence, it is imper- ative to reduce the computational cost and model storage of pre-trained Transformer-based language models in order to popularize their applications in computer systems, especially in edge devices with limited resources. Several works have been developed in the …

Content truncated to 500 words