BioMegatron: Larger Biomedical Domain Language Model

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 509–521, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 509 TernaryBERT: Distillation-aware Ultra-low Bit BERT Wei Zhang⇤, Lu Hou⇤, Yichun Yin⇤, Lifeng Shang, Xiao Chen, Xin Jiang, Qun Liu Huawei Noah’s Ark Lab {zhangwei379, houlu3, yinyichun, shang.lifeng, chen.xiao2, jiang.xin, qun.liu}@huawei.com Abstract Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deploy- ment to resource-constrained devices. In this work, we propose TernaryBERT, which ternar- izes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and em- pirically investigate the ternarization granular- ity of different parts of BERT. Moreover, to re- duce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique (Jiao et al., 2019) in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full- precision model while being 14.9x smaller. 1 Introduction Transformer-based models have shown great power in various natural language processing (NLP) tasks. Pre-trained with gigabytes of unsupervised data, these models usually have hundreds of millions of parameters. For instance, the BERT-base model has 109M parameters, with the model size of 400+MB if represented in 32-bit floating-point format, which is both computation and memory expensive during inference. This poses great challenges for these models to run on resource-constrained devices like cellphones. To alleviate this problem, various meth- ods are proposed to compress these models, like using low-rank approximation (Ma et al., 2019; Lan et al., 2020), weight-sharing (Dehghani et al., 2019; Lan et al., 2020), knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2019), ⇤Authors contribute equally. Figure 1: Model Size vs. MNLI-m Accuracy. Our pro- posed method (red squares) outperforms other BERT compression methods. Details are in Section 4.4. pruning (Michel et al., 2019; Voita et al., 2019; Fan et al., 2019), adaptive depth and/or width (Liu et al., 2020; Hou et al., 2020), and quantization (Zafrir et al., 2019; Shen et al., 2020; Fan et al., 2020). Compared with other compression methods, quantization compresses a neural network by us- ing lower bits for weight values without changing the model architecture, and is particularly useful for carefully-designed network architectures like Transformers. In addition to weight quantization, further quantizing activations can speed up infer- ence with target hardware by turning floating-point operations into integer or bit operations. In (Prato et al., 2019; Zafrir et al., 2019), 8-bit quantization is successfully applied to Transformer-based models with comparable performance as the full-precision baseline. However, quantizing these models to ultra low bits (e.g., 1 or 2 bits) can be much more chal- lenging due to significant reduction in model capac- ity. To avoid severe accuracy drop, more complex quantization methods, like mixed-precision quan- tization (Shen et al., 2020; Zadeh and Moshovos, 2020) …

Content truncated to 500 words