Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT Prakhar Ganesh1∗, Yao Chen1∗, Xin Lou1, Mohammad Ali Khan1, Yin Yang2, Hassan Sajjad3, Preslav Nakov3, Deming Chen4, Marianne Winslett4 1Advanced Digital Sciences Center, Singapore 2College of Science and Engineering, Hamad Bin Khalifa University, Qatar 3Qatar Computing Research Institute, Hamad Bin Khalifa University, Qatar 4University of Illinois at Urbana-Champaign, USA {prakhar.g,yao.chen,lou.xin,mohammad.k}@adsc-create.edu.sg, {yyang,hsajjad,pnakov}@hbku.edu.qa, {dchen,winslett}@illinois.edu Abstract Pre-trained Transformer-based models have achieved state-of-the-art performance for vari- ous Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has at- tracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially pop- ular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compress- ing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research direc- tions for achieving lightweight, accurate, and generic NLP models. 1 Introduction Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization—all these Natural Language Pro- cessing (NLP) tasks benefit from pre-training a large-scale generic model on an enormous corpus such as a Wikipedia dump and/or a book collec- tion, and then fine-tuning for specific downstream tasks, as shown in Figure 1. Earlier solutions fol- lowing this methodology used recurrent neural networks (RNNs) as the base model, for example, ULMFiT (Howard and Ruder, 2018) and ELMo ∗Both authors contributed equally to this research. (Peters et al., 2018), but more recent methods use the Transformer architecture (Vaswani et al., 2017), which relies heavily on the attention mechanism. Popular pre-trained Transformers include BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), XLNet (Yang et al., 2019), MegatronLM (Shoeybi et al., 2019), Turing-NLG (Rosset, 2020), T5 (Raffel et al., 2020), and GPT-3 (Brown et al., 2020). These Transformers are—for exam- ple, BERT, when first released, improved the state of the art for eleven NLP tasks by sizable margins (Devlin et al., 2019). However, Transformers are also bulky and resource-hungry: For instance, GPT-3 (Brown et al., 2020), a recent large-scale Transformer, has over 175 billion parameters. Models of this size incur high memory consump- tion, computational overhead, and energy. The problem is exacerbated when we consider devices with lower capacity (e.g., smartphones), and ap- plications with strict latency constraints, (e.g., interactive chatbots). To put things in perspective, a single training run for GPT-3 (Brown et al., 2020), one of the most powerful and heaviest Transformer-based models, trained on a total of 300 billion tokens, costs well above 12 million USD (Floridi and Chiriatti, 2020). Moreover, fine-tuning or even inference with such a model on a downstream task cannot be done on a GPU with 32GB memory, which is the capacity of Tesla V100, one of the most advanced data center GPUs. …

Content truncated to 500 words