Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6626 - 6638 December 7-11, 2022 ©2022 Association for Computational Linguistics Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters Hongyu Zhao∗ University of Chicago hongyuz@uchicago.edu Hao Tan Adobe Research hatan@adobe.com Hongyuan Mei TTI-Chicago hongyuan@ttic.edu Abstract Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks by adding and tuning a small number of new parameters. Previously proposed adapter architectures are all feed-forward neural net- works. In this paper, we investigate the effec- tiveness of using tiny-attention—i.e., attention with extremely small per-head dimensionality— as adapters. Our tiny-attention adapter learns to modify the hidden states at each position di- rectly conditioned on the hidden states at all the other positions, which is missed by the previ- ously proposed adapters. Moreover, we view its multiple attention heads as a mixture of experts and propose to average their weights during de- ployment, which further reduces its inference computation cost. On the GLUE benchmark, our tiny-attention adapter outperforms the other parameter-efficient transfer learning methods as well as full fine-tuning while only updating 0.05% of the parameters. On the FewGLUE benchmark, its performance is comparable to that of GPT-3 and PET. 1 Introduction Transferring a large pretrained language model (PLM) is a de facto paradigm of performing down- stream tasks in natural language processing (NLP). A general approach is adapter-tuning, which means inserting adapters—i.e., neural networks with small numbers of parameters—into each pretrained layer and only updating the adapter parameters. Adapter-tuning is parameter-efficient and enjoys low computation cost since it keeps the PLM frozen. But it underperforms full fine-tuning which updates all the parameters of the PLM. In this paper, we propose a new adapter architec- ture which outperforms full fine-tuning yet uses even fewer parameters than the previously pro- posed adapters as well as other parameter-efficient transfer learning methods; see Figure 1. ∗Work done while during internship at TTI-Chicago. AdapterP 88.4 AdapterH 87.8 LoRA 88.6 AdapterP 87.9 AdapterH 86.4 Tiny-Attn-1H 89.1 AdaMix 89.9 Finetuning 88.9 avg score Tiny-Attn-4H 89.3 trainable parameters (‰ of all parameters) 0.5 1 2 4 8 16 84 86 87 89 90 85 88 91 0.25 0 AdapterP 88.4 LoRA 88.6 AdapterP 87.9 Tiny-Attn-4H 89.3 Tiny-Attn-1H 89.1 AdapterH 86.4 WARP 84.8 AdaMix 89.9 Finetuning 88.9 AdapterH 87.8 Figure 1: Average performance of different parameter- efficient transfer learning methods on the GLUE bench- mark with roberta-large as the PLM. Our method is Tiny- Attn (with marker): “1H” and “4H” mean “one atten- tion head” and “four attention heads (with the parameter- averaging trick in section 2.2)” respectively. Our adapter is a multi-head attention module and its per-head dimensionality is extremely small; thus we call it tiny-attention. The architecture design is inspired by the following intuitions: 1. For each input sequence, each layer of the lan- guage model produces an embedding for each token in the sequence; see Figure 2a. 2. All the parameter-efficient transfer learning methods learn to modify the embeddings to- wards …

Content truncated to 500 words