BinaryBERT: Pushing the Limit of BERT Quantization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4334–4348 August 1–6, 2021. ©2021 Association for Computational Linguistics 4334 BinaryBERT: Pushing the Limit of BERT Quantization Haoli Bai1, Wei Zhang2, Lu Hou2, Lifeng Shang2, Jing Jin3, Xin Jiang2, Qun Liu2, Michael Lyu1, Irwin King1 1 The Chinese University of Hong Kong 2Huawei Noah’s Ark Lab, 3Huawei Technologies Co., Ltd. {hlbai, lyu, king}@cse.cuhk.edu.hk {zhangwei379, houlu3, shang.lifeng, jinjing12, jiang.xin, qun.liu}@huawei.com Abstract The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular so- lution. In this paper, we propose Binary- BERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which ini- tializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after split- ting. Empirical results show that our Binary- BERT has only a slight performance drop com- pared with the full-precision model while be- ing 24× smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks. 1 Introduction Recent pre-trained language models have achieved remarkable performance improvement in various natural language tasks (Vaswani et al., 2017; De- vlin et al., 2019). However, the improvement generally comes at the cost of increasing model size and computation, which limits the deploy- ment of these huge pre-trained language models to edge devices. Various methods have been re- cently proposed to compress these models, such as knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2020), pruning (Michel et al., 2019; Fan et al., 2019), low-rank approxi- mation (Ma et al., 2019; Lan et al., 2020), weight- sharing (Dehghani et al., 2019; Lan et al., 2020; Huang et al., 2021), dynamic networks with adap- tive depth and/or width (Hou et al., 2020; Xin et al., 2020; Zhou et al., 2020), and quantization (Zafrir (a) MRPC. (b) MNLI-m. Figure 1: Performance of quantized BERT with vary- ing weight bit-widths and 8-bit activation. We report the mean results with standard deviations from 10 seeds on MRPC and 3 seeds on MNLI-m, respectively. et al., 2019; Shen et al., 2020; Fan et al., 2020; Zhang et al., 2020). Among all these model compression approaches, quantization is a popular solution as it does not require designing a smaller model architecture. In- stead, it compresses the model by replacing each 32-bit floating-point parameter with a low-bit fixed- point representation. Existing attempts try to quan- tize pre-trained models (Zafrir et al., 2019; Shen et al., 2020; Fan et al., 2020) to even as low as ternary values (2-bit) with minor performance drop (Zhang et al., 2020). However, none of them achieves the binarization (1-bit). As …

Content truncated to 500 words