Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4812–4826 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4812 Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation Insoo Chung∗ Byeongwook Kim∗ Yoonjung Choi Se Jung Kwon Yongkweon Jeon Baeseong Park Sangha Kim Dongsoo Lee Samsung Research, Seoul, Republic of Korea {insooo.chung, byeonguk.kim, yj0807.choi, sejung0.kwon, dragwon.jeon, bpbs.park, sangha01.kim, dongsoo3.lee}@samsung.com Abstract The deployment of widely used Transformer architecture is challenging because of heavy computation load and memory overhead dur- ing inference, especially when the target de- vice is limited in computational resources such as mobile or edge devices. Quantization is an effective technique to address such challenges. Our analysis shows that for a given number of quantization bits, each block of Transformer contributes to translation quality and inference computations in different manners. Moreover, even inside an embedding block, each word presents vastly different contributions. Cor- respondingly, we propose a mixed precision quantization strategy to represent Transformer weights by an extremely low number of bits (e.g., under 3 bits). For example, for each word in an embedding block, we assign dif- ferent quantization bits based on statistical property. Our quantized Transformer model achieves 11.8× smaller model size than the baseline model, with less than -0.5 BLEU. We achieve 8.3× reduction in run-time memory footprints and 3.5× speed up (Galaxy N10+) such that our proposed compression strategy enables efficient implementation for on-device NMT. 1 Introduction Transformer (Vaswani et al., 2017) is one of the state-of-the-art approaches for Neural Machine Translation (NMT), and hence, being widely ac- cepted. For example, in WMT19 machine trans- lation tasks, it is reported that 80% of submitted systems have adopted the Transformer architec- ture (Barrault et al., 2019). Note that high transla- tion quality of Transformer models entails a large number of parameters. Moreover, the Transformer model is inherently much slower than conventional ∗ Equal Contribution. machine translation approaches (e.g., statistical ap- proaches) mainly due to the auto-regressive infer- ence scheme (Graves, 2013) incrementally gener- ating each token. As a result, deploying the Trans- former model to mobile devices with limited re- sources involves numerous practical implementa- tion issues. To address such implementation challenges with little degradation in translation quality, we study a low-bit quantization strategy for Transformer to accomplish high-performance on-device NMT. We note that most previous studies to compress Transformer models utilize uniform quantization (e.g. INT8 or INT4). While uniform quantiza- tion may be effective for memory footprint savings, it would face various issues to improve inference time and to maintain reasonable BLEU score. For example, even integer arithmetic units for infer- ence operations present limited speed up (Bhandare et al., 2019) and resulting BLEU score of quan- tized Transformer can be substantially degraded with low-bit quantization such as INT4 (Prato et al., 2019). While determining the number of quantization bits for Transformer, it is crucial to consider that each component of Transformer may exhibit varied sensitivity of quantization error toward degrada- tion in translation quality (Wang and Zhang, 2020). Accordingly, a …

Content truncated to 500 words