Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 916–929 May 2-6, 2023 ©2023 Association for Computational Linguistics Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers Minsoo Kim1, Kyuhong Shim2, Seongmin Park1, Wonyong Sung2 and Jungwook Choi1∗ 1Department of Electronic Engineering, Hanyang University 2Department of Electrical and Computer Engineering, Seoul National University {minsoo2333, skstjdals, choij}@hanyang.ac.kr {skhu20, wysung}@snu.ac.kr Abstract Pre-trained Transformer models such as BERT have shown great success in a wide range of applications, but at the cost of substantial in- creases in model complexity. Quantization- aware training (QAT) is a promising method to lower the implementation cost and energy consumption. However, aggressive quantiza- tion below 2-bit causes considerable accuracy degradation due to unstable convergence, espe- cially when the downstream dataset is not abun- dant. This work proposes a proactive knowl- edge distillation method called Teacher Inter- vention (TI) for fast converging QAT of ultra- low precision pre-trained Transformers. TI in- tervenes layer-wise signal propagation with the intact signal from the teacher to remove the interference of propagated quantization errors, smoothing loss surface of QAT and expedit- ing the convergence. Furthermore, we propose a gradual intervention mechanism to stabilize the recovery of subsections of Transformer lay- ers from quantization. The proposed schemes enable fast convergence of QAT and improve the model accuracy regardless of the diverse characteristics of downstream fine-tuning tasks. We demonstrate that TI consistently achieves superior accuracy with significantly lower fine- tuning iterations on well-known Transformers of natural language processing as well as com- puter vision compared to the state-of-the-art QAT methods. 1 Introduction The Transformer-based pre-trained neural networks have significantly improved the performance of various applications of artificial intelligence, in- cluding natural language processing (NLP) (De- vlin et al., 2019; Raffel et al., 2020; Brown et al., 2020) and computer vision (CV) (Dosovitskiy et al., 2020; Touvron et al., 2021; Liu et al., 2021a). The self-attention mechanism represents these models ∗Corresponding Author Figure 1: Comparison of fine-tuning time and accu- racy of ternary weight quantized Transformer models between Teacher Intervention (TI) and TernaryBERT (BERT-base and TinyBERT-6L for CoLA task). TI achieves higher accuracy within 12.5× shorter fine- tuning time. (Vaswani et al., 2017), which links different sym- bols within a sequence to obtain a relational repre- sentation. Thanks to the exceptional performance of the pre-trained Transformer models, there have been increasing needs for their efficient deploy- ment. However, the gigantic size of the pre-trained Transformer models hinders straightforward im- plementation. Even relatively small models like BERT-base (Devlin et al., 2019) contain a few hundred million parameters, incurring profound memory and computation overhead for resource- constrained devices with limited memory and com- puting fabric. Therefore, seminal research efforts attempted to reduce this burden via model compres- sion. (Behnke and Heafield, 2020) and (Gordon et al., 2020) pruned unimportant weights to reduce the number of parameters, while (Mao et al., 2020) further employed low-rank matrix factorization. In addition, Knowledge Distillation (KD) (Hinton et al., 2015) was employed in …

Content truncated to 500 words