On the weak link between importance and prunability of attention heads

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3230–3235, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3230 On the weak link between importance and prunability of attention heads Aakriti Budhraja Madhura Pande Preksha Nema Pratyush Kumar Mitesh M. Khapra Robert Bosch Centre for Data Science and Artificial Intelligence (RBC-DSAI) IIT Madras, India {abudhra,mpande,preksha,pratyush,miteshk}@cse.iitm.ac.in Abstract Given the success of Transformer-based mod- els, two directions of study have emerged: in- terpreting role of individual attention heads and down-sizing the models for efficiency. Our work straddles these two streams: We analyse the importance of basing pruning strategies on the interpreted role of the at- tention heads. We evaluate this on Trans- former and BERT models on multiple NLP tasks. Firstly, we find that a large fraction of the attention heads can be randomly pruned with limited effect on accuracy. Secondly, for Transformers, we find no advantage in prun- ing attention heads identified to be important based on existing studies that relate impor- tance to the location of a head. On the BERT model too we find no preference for top or bottom layers, though the latter are reported to have higher importance. However, strate- gies that avoid pruning middle layers and con- secutive layers perform better. Finally, during fine-tuning the compensation for pruned atten- tion heads is roughly equally distributed across the un-pruned heads. Our results thus suggest that interpretation of attention heads does not strongly inform pruning. 1 Introduction The acclaimed success of Transformer-based mod- els across NLP tasks has been followed by two im- portant directions of research. In the first direction, interpretability studies aim to understand how these models work. Given that multi-headed attention is an important feature of these models, researchers have focused on attention heads as the units of in- terpretation. These studies comment on the role of each attention head and the relation between a head’s position and its significance (Clark et al., 2019; Michel et al., 2019; Voita et al., 2019b,a; Liu et al., 2019; Belinkov et al., 2017). These studies show that certain heads are more important based on (i) their position in the network (top, middle, bottom), or (ii) the component to which they be- long (encoder self-attention, decoder self-attention, encoder-decoder cross attention), or (iii) the func- tional role they play (e.g., syntactic/semantic). In the other major direction, these large Transformer-based models have been down-sized to be more time and space efficient. Different meth- ods for down-sizing have been studied such as prun- ing (McCarley, 2019; Gordon et al., 2020; Sajjad et al., 2020), distillation (Sanh et al., 2019; Liu et al., 2019; Jiao et al., 2019), weight quantiza- tion (Zafrir et al., 2019; Shen et al., 2019), and weight factorization and parameter sharing (Lan et al., 2019). Pruning techniques have been partic- ularly successful in reinforcing the folk-lore that these models are highly over-parameterized. These pruning methods prune parameters based on magni- tude (Gordon et al., 2020), importance (McCarley, 2019) or layer-wise (Sajjad et al., 2020). In …

Content truncated to 500 words