Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns

Findings of the Association for Computational Linguistics: ACL 2022, pages 916 - 925 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns Zihan Wang1∗ Jiuxiang Gu2 Jason Kuen2 Handong Zhao2 Vlad I. Morariu2 Ruiyi Zhang2 Ani Nenkova2 Tong Sun2 Jingbo Shang1 1University of California, San Diego 2Adobe Research 1{ziw224, jshang}@ucsd.edu 2{jigu, kuen, hazhao, morariu, ruizhang, nenkova, tsun}@adobe.com Abstract We present a comprehensive study of sparse at- tention patterns in Transformer models. We first question the need for pre-training with sparse attention and present experiments show- ing that an efficient fine-tuning only approach yields a slightly worse but still competitive model. Then we compare the widely used lo- cal attention pattern and the less-well-studied global attention pattern, demonstrating that global patterns have several unique advantages. We also demonstrate that a flexible approach to attention, with different patterns across differ- ent layers of the model, is beneficial for some tasks. Drawing on this insight, we propose a novel Adaptive Axis Attention method, which learns—during fine-tuning—different attention patterns for each Transformer layer depending on the downstream task. Rather than choosing a fixed attention pattern, the adaptive axis atten- tion method identifies important tokens—for each task and model layer—and focuses atten- tion on those. It does not require pre-training to accommodate the sparse patterns and demon- strates competitive and sometimes better per- formance against fixed sparse attention patterns that require resource-intensive pre-training. 1 Introduction The wide adoption of the Transformer architec- ture (Vaswani et al., 2017) in contextual language representations such as BERT (Devlin et al., 2019) has spurred interest in making transformers more efficient via sparse attention patterns (Li et al., 2019; Guo et al., 2019; Gong et al., 2019; Zaheer et al., 2020; Child et al., 2019). The typical process for learning a transformer model (e.g., BERT) with a sparse attention pattern is to replace the full attention calculation with that pattern, then pre-train the model with the usual pre-training task and fine-tune the model to down- stream tasks. The use of sparse attention pattern ∗ This work was done during the author’s internship at Adobe Research. does not necessarily significantly improve the run time of the models1 but it does reduce the model memory requirement during inference time. This reduction is helpful when deploying models on mo- bile devices or other memory-limited devices. In this paper we offer an extensive analysis of attention patterns, organized around the following questions: (1) is pre-training essential or is it pos- sible to employ sparse patterns during fine-tuning only? (2) which types of attention patterns are important? (3) should the same attention pattern be applied to different downstream tasks and to all layers of the model? The answer to the first question carries critical implications for the practical adoption of sparse attention approaches. Most current transformer- based approaches learn fixed patterns during pre- training and then apply these to fine-tuning as well. However, it is costly and impractical to pre-train a new model from scratch …

Content truncated to 500 words