On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4147–4157 August 1–6, 2021. ©2021 Association for Computational Linguistics 4147 On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers Tianchu Ji1, Shraddhan Jain2, Michael Ferdman2, Peter Milder1, H. Andrew Schwartz2, and Niranjan Balasubramanian2 1,2Stony Brook University 1{tianchu.ji, peter.milder}@stonybrook.edu 2{shrjain, mferdman, has, niranjan}@cs.stonybrook.edu Abstract How much information do NLP tasks really need from a transformer’s attention mecha- nism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accu- racies. However, this requires retraining or even creating entirely new models, both of which can be expensive and carbon-emitting. Focused on optimizations that do not require training, we systematically study the full range of typical attention values necessary. This in- forms the design of an inference-time quanti- zation technique using both pruning and log- scaled mapping which produces only a few (e.g. 23) unique values. Over the tasks of ques- tion answering and sentiment analysis, we find nearly 80% of attention values can be pruned to zeros with minimal (< 1.0%) relative loss in accuracy. We use this pruning technique in conjunction with quantizing the attention val- ues to only a 3-bit format, without retraining, resulting in only a 0.8% accuracy reduction on question answering with fine-tuned RoBERTa. 1 Introduction While the verdict is still out on which large lan- guage model will prove best, at this point in time, all contenders rely on multi-headed attention over multiple layers. Many have investigated whether attention (the output of the softmax, α) itself is qualitatively sensible (e.g., correlating with lin- guistic aspects) (Vig and Belinkov, 2019; Clark et al., 2019; Voita et al., 2018, 2019; Kovaleva et al., 2019; Rogers et al., 2020) or how useful it is for interpreting models (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Brunner et al., 2020; Rogers et al., 2020). Others have focused on in- ducing sparsity in the attention: whether some of the structural components (the softmax function, attention heads and layers) introduce attention spar- sity (Correia et al., 2019; Michel et al., 2019; Voita et al., 2019; Sajjad et al., 2020), if the model tends to focus on a small amount of tokens (Clark et al., 2019; Ramsauer et al., 2020), and the interpretabil- ity of such sparsity (Chen et al., 2020; Rogers et al., 2020). Yet, little is known about our ability to induce sparsity or reduce its values at application- time, and what role the inherent sparsity could play in building inference-time efficient transformers. This work focuses on a systematic study of the quantitative distribution of the attention values across the layers and heads as well as the potential for reducing the information content of attention values during inference at application-time1. We consider two popular pretrained transformer mod- els: BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) over tasks of Masked Language Mod- eling as …

Content truncated to 500 words