Compression of Generative Pre-trained Language Models via Quantization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4821 - 4836 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Compression of Generative Pre-trained Language Models via Quantization Chaofan Tao1, Lu Hou2, Wei Zhang2, Lifeng Shang2, Xin Jiang2, Qun Liu2, Ping Luo1, Ngai Wong1 1The University of Hong Kong 2Huawei Noah’s Ark Lab cftao@connect.hku.hk, pluo@cs.hku.hk, nwong@eee.hku.hk {houlu3, zhangwei379, shang.lifeng, jiang.xin, qun.liu}@huawei.com Abstract The increasing size of generative Pre-trained Language Models (PLMs) have greatly in- creased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to com- press generative PLMs, and the underlying dif- ficulty remains unclear. In this paper, we com- press generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the homogeneous word embeddings caused by reduced capacity, and varied distribution of weights. Correspond- ingly, we propose a token-level contrastive dis- tillation to learn distinguishable word embed- dings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state- of-the-art compression methods on generative PLMs by a clear margin. With comparable per- formance with the full-precision models, we achieve 14.4× and 13.4× compression rates on GPT-2 and BART, respectively. 1 Introduction Transformer-based generative pre-trained language models (PLMs) show strong abilities of multi- task and few-shot learning, and achieve remark- able performances on various tasks (Radford and Narasimhan, 2018; Brown et al., 2020; Lewis et al., 2020; Raffel et al., 2020; Chen et al., 2021). How- ever, they are usually expensive in terms of both computation and memory due to a large number of parameters, and the token-by-token generation process. Many methods have been proposed to compress PLMs, but mostly focus on understand- ing tasks like sentence classification with BERT (Lan et al., 2019; Sun et al., 2020b; Jiao et al., 2020; Shen et al., 2020; Hou et al., 2020). Recent works try to compress GPT-2 using tensor decom- position (Edalati et al., 2021), and knowledge dis- tillation (Song et al., 2020), but the compression Figure 1: Performance of quantized GPT-2 with varying weight bit-widths and 8-bit activation, using different methods. The right figure takes a closer look at LAQ. ratio achieved is much smaller than that of BERT. Yet the underlying difficulty remains unclear. In this paper, we firstly explore compressing gen- erative PLMs by quantizing the parameters from full-precision to lower bits. We find that directly ap- plying previous quantization methods designed for BERT or computer vision tasks to generative PLMs lead to poor performance. Figure 1 shows that the performance drops sharply as the weight bit-width decreases. To investigate the difficulty of quan- tizing generative PLMs, we find that the learned embeddings tend to be homogeneous and hard to distinguish due to the reduced capacity caused by quantization, while the weight distributions also vary significantly across different modules and dif- ferent Transformer layers. These problems are further magnified due …

Content truncated to 500 words