HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8521–8537 July 9-14, 2023 ©2023 Association for Computational Linguistics HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation Anchun Gui and Han Xiao∗ Department of Artificial Intelligence School of Informatics, Xiamen University anchungui@stu.xmu.edu.cn, bookman@xmu.edu.cn Abstract To fully leverage the advantages of large-scale pre-trained language models (PLMs) on down- stream tasks, it has become a ubiquitous adap- tation paradigm to fine-tune the entire param- eters of PLMs. However, this paradigm poses issues of inefficient updating and resource over- consuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs. To alleviate these concerns, in this paper, we propose a parameter- efficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are fine- tuned. To search for those significant attention heads, we develop a novel framework to ana- lyze the effectiveness of heads. Specifically, we first model the relationship between heads into a graph from two perspectives of informa- tion richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head. Extensive experi- ments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines. 1 Introduction Recently large-scale pre-trained language models (PLMs) have triggered a technological revolution in natural language processing (NLP), as the satisfac- tory performance could be achieved by fully fine- tuning parameters of PLMs (Devlin et al., 2019; Brown et al., 2020; Wei et al., 2021). In data- scarce and resource-limited scenarios, however, this methodology poses several concerns. On one hand, full fine-tuning leads to inefficient updating and catastrophic forgetting issues when the train- ing set is insufficient (Houlsby et al., 2019; Wang et al., 2021); on the other hand, this approach has to duplicate a modified copy of full parameters per ∗Corresponding author. (a) Full Fine-tuning (b1) Adapter-like (c) Non-structured Method (b2) HiFi (Ours) (b) Structured Method Updated Param. Extra Updated Param. Frozen Param. Figure 1: Comparison of diverse fine-tuning paradigms. For an example of a set of head weights, full fine-tuning updates all weights, whereas the non-structured meth- ods randomly fine-tune a subset of parameters. For the structured methods, Adapter-like models update the extra weights, while our proposed HiFi selects several vital heads for fine-tuning. task, which presents the challenge of resource over- consuming for the limited storage infrastructure. Parameter-efficient fine-tuning (PEFT), as an al- ternative paradigm, has attracted more attention recently (He et al., 2022; Ding et al., 2022). Com- pared to full fine-tuning, PEFT only fine-tunes the minority of the original parameters (or extra intro- duced parameters) instead of the entire parameters of PLMs, as shown in Fig. 1. Although current PEFTs effectively decrease the proportion of train- able parameters, these methods also lead to vary- ing aspects of concerns. For instance, Adapter (Houlsby et al., 2019) not only breaks the model structure by introducing additional parameters …

Content truncated to 500 words