Understanding and Overcoming the Challenges of Efficient Transformer Quantization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947–7969 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7947 Understanding and Overcoming the Challenges of Efficient Transformer Quantization Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort Qualcomm AI Research∗ {ybond, markusn, tijmen}@qti.qualcomm.com Abstract Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high la- tency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges – namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme – per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. Our source code is available at https://github. com/qualcomm-ai-research/ transformer-quantization. 1 Introduction Recently, transformer architectures have shown re- markable improvement in many Natural Language Processing (NLP) tasks and beyond. Based on the original Transformer (Vaswani et al., 2017), language models pre-trained from large corpora of unlabeled text, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang ∗Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. et al., 2019), Transformer-XL (Dai et al., 2019), GPT family (Radford et al., 2018, 2019; Brown et al., 2020), have become an indispensable build- ing block in modern NLP pipelines. They are also increasingly adopted in other areas, including vi- sion (Carion et al., 2020; Dosovitskiy et al., 2020; Chen et al., 2020; Girdhar et al., 2019) and au- dio (Dong et al., 2018; Child et al., 2019). Despite cutting edge results in many applica- tions, pre-trained transformer-based models are extremely large, sometimes exceeding billions of parameters. Hence, efficient deployment of these models on resource-constrained embedded systems, and even sometimes in data centers, has become an important problem due to high latency and pro- hibitively large memory footprint and energy con- sumption. One effective method to tackle this problem is neural network quantization. Quantization reduces memory consumption by using low-bit precision for weight and activation tensors. Is also reduces inference time, and improves energy efficiency by employing low-bit fixed-point arithmetic instead of floating-point arithmetic (Horowitz, 2014). Quantization, however, is not free. It introduces additional noise in the network that can lead to a drop in the model’s performance. While prior work has demonstrated the feasibility of integer- only inference for computer vision models (Lin et al., 2016; Jacob …

Content truncated to 500 words